[Wed Apr 10 07:40:11 2019][2752129.737059] LNet: Removed LNI 10.0.10.51@o2ib7 [Wed Apr 10 07:40:19 2019][2752137.663432] LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 [Wed Apr 10 07:40:19 2019][2752137.671159] alg: No test for adler32 (adler32-zlib) [Wed Apr 10 07:40:20 2019][2752138.472980] Lustre: Lustre: Build Version: 2.12.0.pl4 [Wed Apr 10 07:40:20 2019][2752138.597993] LNet: Using FastReg for registration [Wed Apr 10 07:40:20 2019][2752138.615908] LNet: Added LNI 10.0.10.51@o2ib7 [8/256/0/180] [Wed Apr 10 07:40:21 2019][2752139.771500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [Wed Apr 10 07:40:22 2019][2752140.089949] Lustre: MGS: Connection restored to 0a29bfc6-ba12-89b3-9a91-fda851440829 (at 0@lo) [Wed Apr 10 07:40:22 2019][2752140.599162] Lustre: MGS: Connection restored to (at 10.8.18.33@o2ib6) [Wed Apr 10 07:40:22 2019][2752140.605866] Lustre: Skipped 50 previous similar messages [Wed Apr 10 07:40:23 2019][2752141.600585] Lustre: MGS: Connection restored to c668e3f4-579f-633e-9693-1c3c32335aef (at 10.9.115.2@o2ib4) [Wed Apr 10 07:40:23 2019][2752141.610456] Lustre: Skipped 99 previous similar messages [Wed Apr 10 07:40:25 2019][2752143.602190] Lustre: MGS: Connection restored to (at 10.8.20.35@o2ib6) [Wed Apr 10 07:40:25 2019][2752143.608897] Lustre: Skipped 213 previous similar messages [Wed Apr 10 07:40:30 2019][2752148.274670] Lustre: MGS: Connection restored to (at 10.9.113.5@o2ib4) [Wed Apr 10 07:40:30 2019][2752148.281378] Lustre: Skipped 235 previous similar messages [Wed Apr 10 07:40:41 2019][2752159.534775] LDISKFS-fs (dm-3): file extents enabled[2752159.536760] LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 [Wed Apr 10 07:40:41 2019] [Wed Apr 10 07:40:41 2019][2752159.546689] , maximum tree depth=5 [Wed Apr 10 07:40:41 2019][2752159.695003] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [Wed Apr 10 07:40:41 2019][2752159.722398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [Wed Apr 10 07:40:41 2019][2752159.949235] Lustre: MGS: Connection restored to (at 10.9.102.45@o2ib4) [Wed Apr 10 07:40:41 2019][2752159.956028] Lustre: Skipped 13 previous similar messages [Wed Apr 10 07:40:42 2019][2752160.187871] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.13.29@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 07:40:42 2019][2752160.205356] LustreError: Skipped 3 previous similar messages [Wed Apr 10 07:40:42 2019][2752160.248487] Lustre: fir-MDT0002: Not available for connect from 10.8.13.27@o2ib6 (not set up) [Wed Apr 10 07:40:42 2019][2752160.257187] Lustre: Skipped 2 previous similar messages [Wed Apr 10 07:40:42 2019][2752160.690194] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.107.5@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 07:40:42 2019][2752160.707670] LustreError: Skipped 28 previous similar messages [Wed Apr 10 07:40:42 2019][2752160.761714] Lustre: fir-MDT0002: Not available for connect from 10.9.104.10@o2ib4 (not set up) [Wed Apr 10 07:40:42 2019][2752160.770510] Lustre: Skipped 26 previous similar messages [Wed Apr 10 07:40:43 2019][2752161.716844] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.23.18@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 07:40:43 2019][2752161.734324] LustreError: Skipped 61 previous similar messages [Wed Apr 10 07:40:43 2019][2752161.795660] Lustre: fir-MDT0002: Not available for connect from 10.8.21.14@o2ib6 (not set up) [Wed Apr 10 07:40:43 2019][2752161.804362] Lustre: Skipped 53 previous similar messages [Wed Apr 10 07:40:45 2019][2752163.721137] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.22.7@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 07:40:45 2019][2752163.738507] LustreError: Skipped 119 previous similar messages [Wed Apr 10 07:40:49 2019][2752167.726967] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.8.12.17@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 07:40:49 2019][2752167.744428] LustreError: Skipped 357 previous similar messages [Wed Apr 10 07:40:57 2019][2752175.846055] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 10.9.114.7@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 07:40:57 2019][2752175.863512] LustreError: Skipped 384 previous similar messages [Wed Apr 10 07:40:58 2019][2752176.815463] Lustre: MGS: Connection restored to (at 10.9.106.47@o2ib4) [Wed Apr 10 07:40:58 2019][2752176.822263] Lustre: Skipped 3 previous similar messages [Wed Apr 10 07:41:13 2019][2752191.855229] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.9.101.53@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 07:41:13 2019][2752191.872768] LustreError: Skipped 1640 previous similar messages [Wed Apr 10 07:41:44 2019][2752222.536345] Lustre: fir-MDT0002: Imperative Recovery not enabled, recovery window 300-900 [Wed Apr 10 07:41:44 2019][2752222.708031] Lustre: fir-MDD0002: changelog on [Wed Apr 10 07:41:44 2019][2752222.716656] Lustre: fir-MDT0002: in recovery but waiting for the first client to connect [Wed Apr 10 07:41:44 2019][2752222.923507] LustreError: 11-0: fir-MDT0002-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [Wed Apr 10 07:41:45 2019][2752223.164178] Lustre: MGS: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) [Wed Apr 10 07:41:45 2019][2752223.172275] Lustre: Skipped 708 previous similar messages [Wed Apr 10 07:41:45 2019][2752223.520003] Lustre: fir-MDT0002: Will be in recovery for at least 5:00, or until 1323 clients reconnect [Wed Apr 10 07:41:46 2019][2752224.339866] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.104.50@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 07:41:46 2019][2752224.357408] LustreError: Skipped 2452 previous similar messages [Wed Apr 10 07:41:46 2019][2752224.457619] Lustre: fir-MDT0000: Will be in recovery for at least 5:00, or until 1323 clients reconnect [Wed Apr 10 07:42:16 2019][2752254.307248] LustreError: 117909:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8b645f6fdd00 x1625533950571968/t0(0) o601->fir-MDT0000-lwp-OST002a_UUID@10.0.10.107@o2ib7:22/0 lens 336/0 e 0 to 0 dl 1554907342 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Apr 10 07:42:16 2019][2752254.822302] LustreError: 117477:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8b645f6fec00 x1625533950572544/t0(0) o601->fir-MDT0000-lwp-OST002a_UUID@10.0.10.107@o2ib7:22/0 lens 336/0 e 0 to 0 dl 1554907342 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Apr 10 07:42:16 2019][2752254.848427] LustreError: 117477:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 18 previous similar messages [Wed Apr 10 07:42:17 2019][2752255.824132] LustreError: 117909:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8b645f6ff500 x1625533950579552/t0(0) o601->fir-MDT0000-lwp-OST002c_UUID@10.0.10.107@o2ib7:23/0 lens 336/0 e 0 to 0 dl 1554907343 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Apr 10 07:42:17 2019][2752255.850251] LustreError: 117909:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 171 previous similar messages [Wed Apr 10 07:42:20 2019][2752258.989748] LustreError: 117477:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8b690eaa6900 x1625533950600944/t0(0) o601->fir-MDT0000-lwp-OST0026_UUID@10.0.10.107@o2ib7:26/0 lens 336/0 e 0 to 0 dl 1554907346 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Apr 10 07:42:20 2019][2752259.015874] LustreError: 117477:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 546 previous similar messages [Wed Apr 10 07:42:26 2019][2752264.270997] LustreError: 117909:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8b645f6ff500 x1625531913755760/t0(0) o601->fir-MDT0000-lwp-OST0013_UUID@10.0.10.104@o2ib7:2/0 lens 336/0 e 0 to 0 dl 1554907352 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Apr 10 07:42:26 2019][2752264.297035] LustreError: 117909:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 303 previous similar messages [Wed Apr 10 07:42:36 2019][2752274.155741] LustreError: 118083:0:(tgt_handler.c:525:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8b875fafdc50 x1625518932537536/t0(0) o601->fir-MDT0000-lwp-OST0027_UUID@10.0.10.108@o2ib7:12/0 lens 336/0 e 0 to 0 dl 1554907362 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Wed Apr 10 07:42:36 2019][2752274.181868] LustreError: 118083:0:(tgt_handler.c:525:tgt_filter_recovery_request()) Skipped 620 previous similar messages [Wed Apr 10 07:42:43 2019][2752282.045091] Lustre: fir-MDT0000: Recovery already passed deadline 0:58, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. [Wed Apr 10 07:42:43 2019][2752282.061628] Lustre: Skipped 3 previous similar messages [Wed Apr 10 07:42:44 2019][2752282.100009] Lustre: fir-MDT0000: Recovery over after 0:57, of 1323 clients 1323 recovered and 0 were evicted. [Wed Apr 10 07:51:10 2019][2752788.142282] Lustre: fir-MDT0002: haven't heard from client b424099b-0e6f-9f1b-e9e1-b67f18b90317 (at 10.9.101.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b763abef000, cur 1554907870 expire 1554907720 last 1554907643 [Wed Apr 10 08:12:45 2019][2754083.620986] Lustre: MGS: Connection restored to (at 10.9.114.7@o2ib4) [Wed Apr 10 08:12:45 2019][2754083.627700] Lustre: Skipped 2762 previous similar messages [Wed Apr 10 08:19:48 2019][2754507.012198] Lustre: MGS: Connection restored to (at 10.9.101.10@o2ib4) [Wed Apr 10 08:19:48 2019][2754507.018997] Lustre: Skipped 2 previous similar messages [Wed Apr 10 09:05:39 2019][2757257.232124] Lustre: fir-MDT0000: haven't heard from client 672fb03f-9053-1684-c128-790aeca874a8 (at 10.9.103.39@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5c8fbf1400, cur 1554912339 expire 1554912189 last 1554912112 [Wed Apr 10 09:05:39 2019][2757257.254085] Lustre: Skipped 5 previous similar messages [Wed Apr 10 09:25:48 2019][2758466.218523] Lustre: 118327:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 09:25:48 2019][2758466.218523] req@ffff8b6c180a7500 x1628638466275120/t146037894706(0) o36->7d8ca85d-8b80-6a23-8fa9-83dca7eb7196@10.8.27.28@o2ib6:22/0 lens 512/2888 e 1 to 0 dl 1554913552 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 09:25:54 2019][2758472.237488] Lustre: fir-MDT0002: Client 7d8ca85d-8b80-6a23-8fa9-83dca7eb7196 (at 10.8.27.28@o2ib6) reconnecting [Wed Apr 10 09:25:54 2019][2758472.237916] Lustre: fir-MDT0002: Connection restored to 0abd045d-746e-a7a6-f1ec-e59a96ee02da (at 10.9.101.64@o2ib4) [Wed Apr 10 09:25:54 2019][2758472.237917] Lustre: Skipped 2 previous similar messages [Wed Apr 10 09:25:54 2019][2758472.263757] Lustre: Skipped 1 previous similar message [Wed Apr 10 09:26:15 2019][2758493.307533] Lustre: fir-MDT0002: Client f4b95b51-aa49-8c42-0e74-2b9d33fd9533 (at 10.9.101.64@o2ib4) reconnecting [Wed Apr 10 09:26:15 2019][2758493.317911] Lustre: fir-MDT0002: Connection restored to 0abd045d-746e-a7a6-f1ec-e59a96ee02da (at 10.9.101.64@o2ib4) [Wed Apr 10 09:26:15 2019][2758493.328548] Lustre: Skipped 1 previous similar message [Wed Apr 10 09:26:36 2019][2758514.396618] Lustre: fir-MDT0002: Client 7d8ca85d-8b80-6a23-8fa9-83dca7eb7196 (at 10.8.27.28@o2ib6) reconnecting [Wed Apr 10 09:26:36 2019][2758514.398036] Lustre: fir-MDT0002: Connection restored to 0abd045d-746e-a7a6-f1ec-e59a96ee02da (at 10.9.101.64@o2ib4) [Wed Apr 10 09:26:36 2019][2758514.417505] Lustre: Skipped 2 previous similar messages [Wed Apr 10 09:26:57 2019][2758535.462178] Lustre: fir-MDT0002: Client f4b95b51-aa49-8c42-0e74-2b9d33fd9533 (at 10.9.101.64@o2ib4) reconnecting [Wed Apr 10 09:26:57 2019][2758535.472556] Lustre: fir-MDT0002: Connection restored to 0abd045d-746e-a7a6-f1ec-e59a96ee02da (at 10.9.101.64@o2ib4) [Wed Apr 10 09:26:57 2019][2758535.483184] Lustre: Skipped 1 previous similar message [Wed Apr 10 09:27:02 2019][2758541.188085] LustreError: 118163:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554913532, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b5e75fe0240/0x53d0f213fda5c5e9 lrc: 3/1,0 mode: --/PR res: [0x2c001073f:0x4c4:0x0].0x0 bits 0x13/0x8 rrc: 15 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118163 timeout: 0 lvb_type: 0 [Wed Apr 10 09:27:03 2019][2758541.188142] LustreError: dumping log to /tmp/lustre-log.1554913622.118420 [Wed Apr 10 09:27:03 2019][2758541.234802] LustreError: 118163:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 09:27:18 2019][2758556.540483] Lustre: fir-MDT0002: Client 7d8ca85d-8b80-6a23-8fa9-83dca7eb7196 (at 10.8.27.28@o2ib6) reconnecting [Wed Apr 10 09:27:18 2019][2758556.550747] Lustre: Skipped 1 previous similar message [Wed Apr 10 09:27:18 2019][2758556.553058] Lustre: fir-MDT0002: Connection restored to 0abd045d-746e-a7a6-f1ec-e59a96ee02da (at 10.9.101.64@o2ib4) [Wed Apr 10 09:27:39 2019][2758577.617032] Lustre: fir-MDT0002: Client f4b95b51-aa49-8c42-0e74-2b9d33fd9533 (at 10.9.101.64@o2ib4) reconnecting [Wed Apr 10 09:28:00 2019][2758598.632424] Lustre: fir-MDT0002: Client 7d8ca85d-8b80-6a23-8fa9-83dca7eb7196 (at 10.8.27.28@o2ib6) reconnecting [Wed Apr 10 09:28:00 2019][2758598.642685] Lustre: Skipped 1 previous similar message [Wed Apr 10 09:28:00 2019][2758598.648028] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.28@o2ib6) [Wed Apr 10 09:28:00 2019][2758598.655434] Lustre: Skipped 3 previous similar messages [Wed Apr 10 09:28:42 2019][2758640.743606] Lustre: fir-MDT0002: Client f4b95b51-aa49-8c42-0e74-2b9d33fd9533 (at 10.9.101.64@o2ib4) reconnecting [Wed Apr 10 09:28:42 2019][2758640.753953] Lustre: Skipped 4 previous similar messages [Wed Apr 10 09:28:53 2019][2758651.310438] LNet: Service thread pid 118420 was inactive for 200.12s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 09:28:53 2019][2758651.327642] Pid: 118420, comm: mdt01_106 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 09:28:53 2019][2758651.337637] Call Trace: [Wed Apr 10 09:28:53 2019][2758651.340282] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.347391] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.354771] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 09:28:53 2019][2758651.361703] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 09:28:53 2019][2758651.368373] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 09:28:53 2019][2758651.374598] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 09:28:53 2019][2758651.381356] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 09:28:53 2019][2758651.387234] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.394364] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.402252] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.408770] [] kthread+0xd1/0xe0 [Wed Apr 10 09:28:53 2019][2758651.413857] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 09:28:53 2019][2758651.420535] [] 0xffffffffffffffff [Wed Apr 10 09:28:53 2019][2758651.425735] LustreError: dumping log to /tmp/lustre-log.1554913733.118420 [Wed Apr 10 09:28:53 2019][2758651.625031] Pid: 118163, comm: mdt01_015 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 09:28:53 2019][2758651.635031] Call Trace: [Wed Apr 10 09:28:53 2019][2758651.637676] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.644795] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.652177] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 09:28:53 2019][2758651.659182] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 09:28:53 2019][2758651.666372] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 09:28:53 2019][2758651.673050] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 09:28:53 2019][2758651.680240] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 09:28:53 2019][2758651.686988] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 09:28:53 2019][2758651.693667] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.700604] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.707927] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.714262] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.721391] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.729281] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 09:28:53 2019][2758651.735779] [] kthread+0xd1/0xe0 [Wed Apr 10 09:28:53 2019][2758651.740869] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 09:28:53 2019][2758651.747531] [] 0xffffffffffffffff [Wed Apr 10 09:29:05 2019][2758663.261789] Lustre: fir-MDT0000: haven't heard from client 26e9285d-a17e-4504-ddeb-2f45459e682f (at 10.8.23.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b58d7a5b400, cur 1554913745 expire 1554913595 last 1554913518 [Wed Apr 10 09:29:05 2019][2758663.283663] Lustre: Skipped 2 previous similar messages [Wed Apr 10 09:29:22 2019][2758680.262084] Lustre: fir-MDT0002: haven't heard from client 26e9285d-a17e-4504-ddeb-2f45459e682f (at 10.8.23.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b86c9b36800, cur 1554913762 expire 1554913612 last 1554913535 [Wed Apr 10 09:29:24 2019][2758682.753104] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.28@o2ib6) [Wed Apr 10 09:29:24 2019][2758682.760505] Lustre: Skipped 10 previous similar messages [Wed Apr 10 09:30:06 2019][2758724.849458] Lustre: fir-MDT0002: Client f4b95b51-aa49-8c42-0e74-2b9d33fd9533 (at 10.9.101.64@o2ib4) reconnecting [Wed Apr 10 09:30:06 2019][2758724.859809] Lustre: Skipped 7 previous similar messages [Wed Apr 10 09:31:01 2019][2758779.595157] Lustre: 117491:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 09:31:01 2019][2758779.595157] req@ffff8b538501cb00 x1629298901033872/t146038867267(0) o36->40c21896-2b28-d1d6-5611-05f09dc83863@10.9.108.24@o2ib4:6/0 lens 512/2888 e 1 to 0 dl 1554913866 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 09:31:01 2019][2758779.625223] Lustre: 117491:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 09:31:11 2019][2758789.607363] Lustre: 117530:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 09:31:11 2019][2758789.607363] req@ffff8b7a5ca9a100 x1628719134381568/t0(0) o101->ecfe048c-31b4-368b-947d-642490fbb46b@10.9.105.41@o2ib4:16/0 lens 576/3264 e 0 to 0 dl 1554913876 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 09:31:48 2019][2758826.667034] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 09:31:48 2019][2758826.674524] Lustre: Skipped 15 previous similar messages [Wed Apr 10 09:32:16 2019][2758854.545701] LustreError: 117557:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554913846, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b6bad58ba80/0x53d0f21400384f6c lrc: 3/1,0 mode: --/PR res: [0x2c00111af:0x536:0x0].0x0 bits 0x13/0x8 rrc: 16 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 117557 timeout: 0 lvb_type: 0 [Wed Apr 10 09:32:16 2019][2758854.585470] LustreError: 117557:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 09:32:19 2019][2758857.744987] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Wed Apr 10 09:32:19 2019][2758857.755338] Lustre: Skipped 18 previous similar messages [Wed Apr 10 09:34:06 2019][2758964.660986] LNet: Service thread pid 117557 was inactive for 200.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 09:34:06 2019][2758964.678189] LNet: Skipped 1 previous similar message [Wed Apr 10 09:34:06 2019][2758964.683339] Pid: 117557, comm: mdt01_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 09:34:06 2019][2758964.693340] Call Trace: [Wed Apr 10 09:34:06 2019][2758964.695984] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 09:34:06 2019][2758964.703119] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 09:34:06 2019][2758964.710489] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 09:34:06 2019][2758964.717522] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 09:34:06 2019][2758964.724701] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 09:34:06 2019][2758964.731373] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 09:34:06 2019][2758964.738587] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 09:34:06 2019][2758964.745346] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 09:34:06 2019][2758964.751994] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 09:34:06 2019][2758964.758942] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 09:34:06 2019][2758964.766224] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 09:34:06 2019][2758964.772569] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 09:34:06 2019][2758964.779684] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 09:34:06 2019][2758964.787596] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 09:34:06 2019][2758964.794097] [] kthread+0xd1/0xe0 [Wed Apr 10 09:34:06 2019][2758964.799183] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 09:34:06 2019][2758964.805833] [] 0xffffffffffffffff [Wed Apr 10 09:34:06 2019][2758964.811035] LustreError: dumping log to /tmp/lustre-log.1554914046.117557 [Wed Apr 10 09:34:07 2019][2758965.451739] Pid: 118208, comm: mdt00_024 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 09:34:07 2019][2758965.461763] Call Trace: [Wed Apr 10 09:34:07 2019][2758965.464413] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 09:34:07 2019][2758965.471553] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 09:34:07 2019][2758965.478935] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 09:34:07 2019][2758965.485853] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 09:34:07 2019][2758965.492524] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 09:34:07 2019][2758965.498749] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 09:34:07 2019][2758965.505522] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 09:34:07 2019][2758965.511408] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 09:34:07 2019][2758965.518563] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 09:34:07 2019][2758965.526456] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 09:34:07 2019][2758965.532956] [] kthread+0xd1/0xe0 [Wed Apr 10 09:34:07 2019][2758965.538043] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 09:34:07 2019][2758965.544691] [] 0xffffffffffffffff [Wed Apr 10 09:36:22 2019][2759100.999978] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.24@o2ib4) [Wed Apr 10 09:36:22 2019][2759101.007470] Lustre: Skipped 47 previous similar messages [Wed Apr 10 09:36:43 2019][2759122.014496] Lustre: fir-MDT0002: Client 40c21896-2b28-d1d6-5611-05f09dc83863 (at 10.9.108.24@o2ib4) reconnecting [Wed Apr 10 09:36:43 2019][2759122.024849] Lustre: Skipped 44 previous similar messages [Wed Apr 10 09:36:51 2019][2759130.235383] LustreError: 118646:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001212a:0x2bd:0x0] [Wed Apr 10 09:45:07 2019][2759626.176220] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.24@o2ib4) [Wed Apr 10 09:45:07 2019][2759626.183712] Lustre: Skipped 91 previous similar messages [Wed Apr 10 09:45:28 2019][2759647.189486] Lustre: fir-MDT0002: Client 40c21896-2b28-d1d6-5611-05f09dc83863 (at 10.9.108.24@o2ib4) reconnecting [Wed Apr 10 09:45:28 2019][2759647.199831] Lustre: Skipped 91 previous similar messages [Wed Apr 10 09:48:02 2019][2759800.285478] Lustre: fir-MDT0000: haven't heard from client d681b0c6-fb2c-bc67-5eb0-adc9739bfde6 (at 10.9.114.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5902401800, cur 1554914882 expire 1554914732 last 1554914655 [Wed Apr 10 09:48:02 2019][2759800.307382] Lustre: Skipped 1 previous similar message [Wed Apr 10 09:55:17 2019][2760235.353531] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.24@o2ib4) [Wed Apr 10 09:55:17 2019][2760235.361047] Lustre: Skipped 106 previous similar messages [Wed Apr 10 09:55:35 2019][2760253.431803] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Wed Apr 10 09:55:35 2019][2760253.442149] Lustre: Skipped 105 previous similar messages [Wed Apr 10 10:05:24 2019][2760842.529897] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 10:05:24 2019][2760842.537384] Lustre: Skipped 108 previous similar messages [Wed Apr 10 10:05:38 2019][2760856.474183] Lustre: 118354:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 10:05:38 2019][2760856.474183] req@ffff8b77803d7b00 x1629749577593744/t0(0) o101->a132911f-e28d-2e04-1263-47066608ec3c@10.8.23.24@o2ib6:13/0 lens 576/3264 e 1 to 0 dl 1554915943 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 10:05:38 2019][2760856.503363] Lustre: 118354:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 10:05:44 2019][2760862.482508] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 10:05:44 2019][2760862.492863] Lustre: Skipped 106 previous similar messages [Wed Apr 10 10:06:53 2019][2760931.434729] LustreError: 117530:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554915923, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b6b9a6545c0/0x53d0f2140e127646 lrc: 3/1,0 mode: --/PR res: [0x2c0012902:0x6e:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 117530 timeout: 0 lvb_type: 0 [Wed Apr 10 10:06:53 2019][2760931.474313] LustreError: 117530:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 10:08:43 2019][2761041.887994] LNet: Service thread pid 117530 was inactive for 200.45s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 10:08:43 2019][2761041.905191] LNet: Skipped 1 previous similar message [Wed Apr 10 10:08:43 2019][2761041.910353] Pid: 117530, comm: mdt01_003 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 10:08:43 2019][2761041.920377] Call Trace: [Wed Apr 10 10:08:43 2019][2761041.923021] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 10:08:43 2019][2761041.930165] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 10:08:43 2019][2761041.937533] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 10:08:43 2019][2761041.944555] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 10:08:43 2019][2761041.951732] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 10:08:43 2019][2761041.958388] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 10:08:43 2019][2761041.965690] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 10:08:43 2019][2761041.972516] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 10:08:43 2019][2761041.979232] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 10:08:43 2019][2761041.986298] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 10:08:43 2019][2761041.993643] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 10:08:43 2019][2761041.999990] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 10:08:43 2019][2761042.007161] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 10:08:43 2019][2761042.015053] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 10:08:43 2019][2761042.021587] [] kthread+0xd1/0xe0 [Wed Apr 10 10:08:43 2019][2761042.026674] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 10:08:43 2019][2761042.033337] [] 0xffffffffffffffff [Wed Apr 10 10:08:43 2019][2761042.038531] LustreError: dumping log to /tmp/lustre-log.1554916123.117530 [Wed Apr 10 10:08:45 2019][2761043.336281] LNet: Service thread pid 118220 was inactive for 201.90s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 10:08:45 2019][2761043.353481] Pid: 118220, comm: mdt00_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 10:08:45 2019][2761043.363483] Call Trace: [Wed Apr 10 10:08:45 2019][2761043.366124] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 10:08:45 2019][2761043.373245] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 10:08:45 2019][2761043.380629] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 10:08:45 2019][2761043.387557] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 10:08:45 2019][2761043.394255] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 10:08:45 2019][2761043.400477] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 10:08:45 2019][2761043.407220] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 10:08:45 2019][2761043.413096] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 10:08:45 2019][2761043.420221] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 10:08:45 2019][2761043.428106] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 10:08:45 2019][2761043.434606] [] kthread+0xd1/0xe0 [Wed Apr 10 10:08:45 2019][2761043.439713] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 10:08:45 2019][2761043.446359] [] 0xffffffffffffffff [Wed Apr 10 10:11:03 2019][2761181.988613] LustreError: 118652:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128e6:0x7a90:0x0] [Wed Apr 10 10:14:34 2019][2761392.495105] Lustre: 118489:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 10:14:34 2019][2761392.495105] req@ffff8b6e6675f200 x1628870323751552/t146045153882(0) o36->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:9/0 lens 504/2888 e 1 to 0 dl 1554916479 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 10:15:32 2019][2761451.127530] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Wed Apr 10 10:15:32 2019][2761451.134936] Lustre: Skipped 171 previous similar messages [Wed Apr 10 10:15:44 2019][2761462.643547] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Wed Apr 10 10:15:44 2019][2761462.653900] Lustre: Skipped 171 previous similar messages [Wed Apr 10 10:15:49 2019][2761467.465636] LustreError: 118514:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554916459, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b797c8198c0/0x53d0f214129a05b0 lrc: 3/1,0 mode: --/PR res: [0x2c0010945:0x1c52:0x0].0x0 bits 0x13/0x8 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118514 timeout: 0 lvb_type: 0 [Wed Apr 10 10:15:49 2019][2761467.505386] LustreError: 118514:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 10:17:06 2019][2761544.342235] Lustre: MGS: haven't heard from client 6cd7cd5a-251d-379f-3600-fec1a17f21f2 (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5359390800, cur 1554916626 expire 1554916476 last 1554916399 [Wed Apr 10 10:17:06 2019][2761544.363420] Lustre: Skipped 2 previous similar messages [Wed Apr 10 10:17:21 2019][2761559.321847] Lustre: fir-MDT0002: haven't heard from client 6ca6652e-0187-99bc-e850-704338e05c6f (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7414fc7800, cur 1554916641 expire 1554916491 last 1554916414 [Wed Apr 10 10:17:25 2019][2761563.321612] Lustre: fir-MDT0000: haven't heard from client 6ca6652e-0187-99bc-e850-704338e05c6f (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b501e12e000, cur 1554916645 expire 1554916495 last 1554916418 [Wed Apr 10 10:17:39 2019][2761577.962891] LNet: Service thread pid 118514 was inactive for 200.49s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 10:17:39 2019][2761577.980090] Pid: 118514, comm: mdt02_082 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 10:17:39 2019][2761577.990089] Call Trace: [Wed Apr 10 10:17:39 2019][2761577.992732] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 10:17:39 2019][2761577.999871] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 10:17:39 2019][2761578.007284] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 10:17:39 2019][2761578.014316] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 10:17:39 2019][2761578.021495] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 10:17:39 2019][2761578.028159] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 10:17:39 2019][2761578.035345] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 10:17:39 2019][2761578.042096] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 10:17:39 2019][2761578.048762] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 10:17:39 2019][2761578.055710] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 10:17:39 2019][2761578.063015] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 10:17:39 2019][2761578.069361] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 10:17:39 2019][2761578.076499] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 10:17:39 2019][2761578.084400] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 10:17:39 2019][2761578.090931] [] kthread+0xd1/0xe0 [Wed Apr 10 10:17:39 2019][2761578.096030] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 10:17:39 2019][2761578.102691] [] 0xffffffffffffffff [Wed Apr 10 10:17:39 2019][2761578.107898] LustreError: dumping log to /tmp/lustre-log.1554916659.118514 [Wed Apr 10 10:17:40 2019][2761578.933756] Pid: 118174, comm: mdt02_012 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 10:17:40 2019][2761578.943752] Call Trace: [Wed Apr 10 10:17:40 2019][2761578.946389] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 10:17:40 2019][2761578.953518] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 10:17:40 2019][2761578.960907] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 10:17:40 2019][2761578.967836] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 10:17:40 2019][2761578.974515] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 10:17:40 2019][2761578.980740] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 10:17:40 2019][2761578.987498] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 10:17:40 2019][2761578.993377] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 10:17:40 2019][2761579.000516] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 10:17:40 2019][2761579.008406] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 10:17:40 2019][2761579.014920] [] kthread+0xd1/0xe0 [Wed Apr 10 10:17:40 2019][2761579.020027] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 10:17:40 2019][2761579.026727] [] 0xffffffffffffffff [Wed Apr 10 10:25:33 2019][2762051.758464] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 10:25:33 2019][2762051.765956] Lustre: Skipped 229 previous similar messages [Wed Apr 10 10:25:44 2019][2762062.737482] Lustre: fir-MDT0002: Client 40c21896-2b28-d1d6-5611-05f09dc83863 (at 10.9.108.24@o2ib4) reconnecting [Wed Apr 10 10:25:44 2019][2762062.747853] Lustre: Skipped 217 previous similar messages [Wed Apr 10 10:27:37 2019][2762175.333956] Lustre: fir-MDT0000: haven't heard from client 60436878-76bd-e2ae-c7ae-914483a7aa01 (at 10.8.21.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b888f213400, cur 1554917257 expire 1554917107 last 1554917030 [Wed Apr 10 10:27:48 2019][2762186.334183] Lustre: MGS: haven't heard from client 8361a3a7-2b72-2bc8-7ab9-7c23c9b37fa2 (at 10.8.21.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6cfb262800, cur 1554917268 expire 1554917118 last 1554917041 [Wed Apr 10 10:35:34 2019][2762652.596592] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.28@o2ib6) [Wed Apr 10 10:35:34 2019][2762652.603994] Lustre: Skipped 218 previous similar messages [Wed Apr 10 10:35:51 2019][2762669.399452] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 10:35:51 2019][2762669.409806] Lustre: Skipped 219 previous similar messages [Wed Apr 10 10:45:39 2019][2763257.445458] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Wed Apr 10 10:45:39 2019][2763257.452871] Lustre: Skipped 218 previous similar messages [Wed Apr 10 10:46:00 2019][2763278.527681] Lustre: fir-MDT0002: Client a132911f-e28d-2e04-1263-47066608ec3c (at 10.8.23.24@o2ib6) reconnecting [Wed Apr 10 10:46:00 2019][2763278.537947] Lustre: Skipped 223 previous similar messages [Wed Apr 10 10:55:48 2019][2763866.552990] Lustre: fir-MDT0002: Connection restored to (at 10.9.103.11@o2ib4) [Wed Apr 10 10:55:48 2019][2763866.560478] Lustre: Skipped 222 previous similar messages [Wed Apr 10 10:56:02 2019][2763881.055523] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Wed Apr 10 10:56:02 2019][2763881.065880] Lustre: Skipped 220 previous similar messages [Wed Apr 10 10:57:56 2019][2763994.371914] Lustre: fir-MDT0000: haven't heard from client e0982516-a138-a328-edef-e2447d7c8df2 (at 10.9.101.40@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5c8c77e000, cur 1554919076 expire 1554918926 last 1554918849 [Wed Apr 10 10:57:56 2019][2763994.393875] Lustre: Skipped 1 previous similar message [Wed Apr 10 11:05:51 2019][2764470.127954] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 11:05:51 2019][2764470.135438] Lustre: Skipped 224 previous similar messages [Wed Apr 10 11:06:08 2019][2764486.447261] Lustre: fir-MDT0002: Client 6ea7ad88-4c41-3a56-d580-e39b0e85132b (at 10.8.27.6@o2ib6) reconnecting [Wed Apr 10 11:06:08 2019][2764486.457438] Lustre: Skipped 220 previous similar messages [Wed Apr 10 11:11:16 2019][2764794.357169] LustreError: 118703:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011155:0x7bd7:0x0] [Wed Apr 10 11:11:50 2019][2764829.079604] Lustre: 118431:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 11:11:50 2019][2764829.079604] req@ffff8b6e75e8fb00 x1629296639628944/t146057716696(0) o36->35ddd205-c174-6b95-20ef-c38a3193733f@10.9.108.35@o2ib4:25/0 lens 544/2888 e 1 to 0 dl 1554919915 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:11:50 2019][2764829.109746] Lustre: 118431:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 11:13:05 2019][2764904.047190] LustreError: 118319:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554919895, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b766fbc1440/0x53d0f21437aba7c0 lrc: 3/1,0 mode: --/PR res: [0x2c00111b8:0x6aa:0x0].0x0 bits 0x13/0x8 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118319 timeout: 0 lvb_type: 0 [Wed Apr 10 11:13:05 2019][2764904.086952] LustreError: 118319:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 11:14:03 2019][2764961.392411] Lustre: fir-MDT0000: haven't heard from client d3813457-4417-03b3-a468-74026cb0ff17 (at 10.9.101.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4cca2b2400, cur 1554920043 expire 1554919893 last 1554919816 [Wed Apr 10 11:14:03 2019][2764961.414287] Lustre: Skipped 2 previous similar messages [Wed Apr 10 11:14:56 2019][2765014.578520] LNet: Service thread pid 118319 was inactive for 200.52s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 11:14:56 2019][2765014.595712] LNet: Skipped 1 previous similar message [Wed Apr 10 11:14:56 2019][2765014.600863] Pid: 118319, comm: mdt02_039 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:14:56 2019][2765014.610886] Call Trace: [Wed Apr 10 11:14:56 2019][2765014.613529] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:14:56 2019][2765014.620666] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:14:56 2019][2765014.628038] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 11:14:56 2019][2765014.635049] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 11:14:56 2019][2765014.642248] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 11:14:56 2019][2765014.648907] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 11:14:56 2019][2765014.656112] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 11:14:56 2019][2765014.662865] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 11:14:56 2019][2765014.669540] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 11:14:56 2019][2765014.676485] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 11:14:56 2019][2765014.683780] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 11:14:56 2019][2765014.690126] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:14:56 2019][2765014.697257] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:14:56 2019][2765014.705155] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:14:56 2019][2765014.711678] [] kthread+0xd1/0xe0 [Wed Apr 10 11:14:56 2019][2765014.716770] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:14:56 2019][2765014.723426] [] 0xffffffffffffffff [Wed Apr 10 11:14:56 2019][2765014.728651] LustreError: dumping log to /tmp/lustre-log.1554920096.118319 [Wed Apr 10 11:14:57 2019][2765016.053009] LNet: Service thread pid 118472 was inactive for 202.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 11:14:57 2019][2765016.070246] Pid: 118472, comm: mdt02_065 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:14:57 2019][2765016.080268] Call Trace: [Wed Apr 10 11:14:57 2019][2765016.082921] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:14:57 2019][2765016.090064] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:14:57 2019][2765016.097450] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 11:14:57 2019][2765016.104396] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 11:14:57 2019][2765016.111079] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 11:14:57 2019][2765016.117329] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 11:14:57 2019][2765016.124081] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 11:14:57 2019][2765016.129978] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:14:57 2019][2765016.137115] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:14:57 2019][2765016.145016] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:14:57 2019][2765016.151527] [] kthread+0xd1/0xe0 [Wed Apr 10 11:14:57 2019][2765016.156637] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:14:57 2019][2765016.163290] [] 0xffffffffffffffff [Wed Apr 10 11:15:55 2019][2765073.761878] LustreError: 118750:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010458:0x8a0:0x0] [Wed Apr 10 11:15:56 2019][2765074.505687] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.6@o2ib6) [Wed Apr 10 11:15:56 2019][2765074.513021] Lustre: Skipped 244 previous similar messages [Wed Apr 10 11:16:08 2019][2765087.297297] Lustre: fir-MDT0002: Client 40c21896-2b28-d1d6-5611-05f09dc83863 (at 10.9.108.24@o2ib4) reconnecting [Wed Apr 10 11:16:08 2019][2765087.307655] Lustre: Skipped 242 previous similar messages [Wed Apr 10 11:16:52 2019][2765130.458961] Lustre: 118392:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 11:16:52 2019][2765130.458961] req@ffff8b79b9ad8c00 x1628639091807552/t0(0) o101->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:27/0 lens 576/3264 e 1 to 0 dl 1554920217 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:16:52 2019][2765130.488238] Lustre: 118392:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 11:17:02 2019][2765140.477181] Lustre: 117474:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 11:17:02 2019][2765140.477181] req@ffff8b86047f3600 x1629772791678992/t146059998755(0) o36->563d53a1-cc0d-a13f-4381-648a7da03caa@10.9.105.60@o2ib4:7/0 lens 512/2888 e 0 to 0 dl 1554920227 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:17:46 2019][2765184.511353] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Wed Apr 10 11:18:07 2019][2765205.428527] LustreError: 118204:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554920197, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b77b7382d00/0x53d0f2143c6da32f lrc: 3/1,0 mode: --/PR res: [0x2c0011928:0x3bc:0x0].0x0 bits 0x13/0x8 rrc: 11 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118204 timeout: 0 lvb_type: 0 [Wed Apr 10 11:18:07 2019][2765205.468288] LustreError: 118204:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 11:19:57 2019][2765315.640839] LNet: Service thread pid 118222 was inactive for 200.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 11:19:57 2019][2765315.658066] Pid: 118222, comm: mdt03_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:19:57 2019][2765315.668070] Call Trace: [Wed Apr 10 11:19:57 2019][2765315.670712] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:19:57 2019][2765315.677869] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:19:57 2019][2765315.685252] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 11:19:57 2019][2765315.692195] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 11:19:57 2019][2765315.698851] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 11:19:57 2019][2765315.705090] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 11:19:57 2019][2765315.711837] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 11:19:57 2019][2765315.717728] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:19:57 2019][2765315.724868] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:19:57 2019][2765315.732781] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:19:57 2019][2765315.739323] [] kthread+0xd1/0xe0 [Wed Apr 10 11:19:57 2019][2765315.744422] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:19:57 2019][2765315.751186] [] 0xffffffffffffffff [Wed Apr 10 11:19:57 2019][2765315.756481] LustreError: dumping log to /tmp/lustre-log.1554920397.118222 [Wed Apr 10 11:19:58 2019][2765316.821008] Pid: 118204, comm: mdt02_020 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:19:58 2019][2765316.831014] Call Trace: [Wed Apr 10 11:19:58 2019][2765316.833662] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:19:58 2019][2765316.840796] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:19:58 2019][2765316.848177] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 11:19:58 2019][2765316.855199] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 11:19:58 2019][2765316.862395] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 11:19:58 2019][2765316.869063] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 11:19:58 2019][2765316.876250] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 11:19:58 2019][2765316.883028] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 11:19:58 2019][2765316.889693] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 11:19:58 2019][2765316.896715] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 11:19:58 2019][2765316.904011] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 11:19:58 2019][2765316.910362] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:19:58 2019][2765316.917495] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:19:58 2019][2765316.925419] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:19:58 2019][2765316.931933] [] kthread+0xd1/0xe0 [Wed Apr 10 11:19:58 2019][2765316.937030] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:19:58 2019][2765316.943695] [] 0xffffffffffffffff [Wed Apr 10 11:20:07 2019][2765325.815770] LustreError: 119191:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b792a167b00 x1628639092324736/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:13/0 lens 448/440 e 0 to 0 dl 1554920413 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:20:27 2019][2765345.510474] Lustre: 118704:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 11:20:27 2019][2765345.510474] req@ffff8b897824e900 x1629293752868912/t146061860713(0) o36->f53d399b-a6b5-36d0-034f-8c8c0efbb65f@10.9.106.11@o2ib4:2/0 lens 512/2888 e 1 to 0 dl 1554920432 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:20:37 2019][2765355.539686] Lustre: 118385:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 11:20:37 2019][2765355.539686] req@ffff8b87dda81500 x1629298190976512/t0(0) o101->ccbe839d-1880-0660-8c5e-d6ce0f68a97f@10.9.108.31@o2ib4:12/0 lens 576/3264 e 0 to 0 dl 1554920442 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:20:37 2019][2765355.569017] Lustre: 118385:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 11:20:38 2019][2765356.770720] LustreError: 118671:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001082f:0x1614:0x0] [Wed Apr 10 11:21:42 2019][2765420.463051] LustreError: 118571:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554920412, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b8ba9488240/0x53d0f2144009cb5a lrc: 3/1,0 mode: --/PR res: [0x2c001118d:0x6e3:0x0].0x0 bits 0x13/0x8 rrc: 19 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118571 timeout: 0 lvb_type: 0 [Wed Apr 10 11:21:42 2019][2765420.502789] LustreError: 118571:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 11:23:32 2019][2765530.685368] LNet: Service thread pid 118305 was inactive for 200.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 11:23:32 2019][2765530.702585] LNet: Skipped 1 previous similar message [Wed Apr 10 11:23:32 2019][2765530.707733] Pid: 118305, comm: mdt00_057 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:23:32 2019][2765530.717752] Call Trace: [Wed Apr 10 11:23:32 2019][2765530.720394] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:23:32 2019][2765530.727512] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:23:32 2019][2765530.734888] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 11:23:32 2019][2765530.741907] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 11:23:32 2019][2765530.749086] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Wed Apr 10 11:23:32 2019][2765530.756275] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 11:23:32 2019][2765530.763023] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 11:23:32 2019][2765530.769679] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 11:23:32 2019][2765530.776651] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 11:23:32 2019][2765530.783934] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 11:23:32 2019][2765530.790287] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:23:32 2019][2765530.797419] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:23:32 2019][2765530.805351] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:23:32 2019][2765530.811865] [] kthread+0xd1/0xe0 [Wed Apr 10 11:23:32 2019][2765530.816956] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:23:32 2019][2765530.823618] [] 0xffffffffffffffff [Wed Apr 10 11:23:32 2019][2765530.828815] LustreError: dumping log to /tmp/lustre-log.1554920612.118305 [Wed Apr 10 11:23:33 2019][2765531.563247] Pid: 117474, comm: mdt03_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:23:33 2019][2765531.573247] Call Trace: [Wed Apr 10 11:23:33 2019][2765531.575919] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.583047] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.590413] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 11:23:33 2019][2765531.597338] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 11:23:33 2019][2765531.603993] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 11:23:33 2019][2765531.610216] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 11:23:33 2019][2765531.616958] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 11:23:33 2019][2765531.622835] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.629960] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.637855] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.644371] [] kthread+0xd1/0xe0 [Wed Apr 10 11:23:33 2019][2765531.649460] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:23:33 2019][2765531.656107] [] 0xffffffffffffffff [Wed Apr 10 11:23:33 2019][2765531.661303] Pid: 118571, comm: mdt03_059 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:23:33 2019][2765531.671303] Call Trace: [Wed Apr 10 11:23:33 2019][2765531.673942] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.681049] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.688444] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 11:23:33 2019][2765531.695448] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 11:23:33 2019][2765531.702623] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 11:23:33 2019][2765531.709279] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 11:23:33 2019][2765531.716457] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 11:23:33 2019][2765531.723198] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 11:23:33 2019][2765531.729853] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.736797] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.744087] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.750451] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.757587] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.765501] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:23:33 2019][2765531.772010] [] kthread+0xd1/0xe0 [Wed Apr 10 11:23:33 2019][2765531.777115] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:23:33 2019][2765531.783761] [] 0xffffffffffffffff [Wed Apr 10 11:25:57 2019][2765675.414513] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.24@o2ib4) [Wed Apr 10 11:25:57 2019][2765675.422004] Lustre: Skipped 364 previous similar messages [Wed Apr 10 11:26:09 2019][2765687.968342] Lustre: fir-MDT0002: Client f53d399b-a6b5-36d0-034f-8c8c0efbb65f (at 10.9.106.11@o2ib4) reconnecting [Wed Apr 10 11:26:09 2019][2765687.978700] Lustre: Skipped 366 previous similar messages [Wed Apr 10 11:30:49 2019][2765967.757624] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554921042/real 1554921042] req@ffff8b4eaaf61b00 x1630438570995456/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554921049 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 11:30:56 2019][2765974.784768] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554921049/real 1554921049] req@ffff8b4eaaf61b00 x1630438570995456/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554921056 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 11:30:56 2019][2765974.812297] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Wed Apr 10 11:30:57 2019][2765975.798787] Lustre: 118430:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 11:30:57 2019][2765975.798787] req@ffff8b56b33bc800 x1628794161426000/t0(0) o101->6005ff27-25fe-bad1-bbec-2de2fb085c90@10.9.112.7@o2ib4:2/0 lens 480/568 e 1 to 0 dl 1554921062 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:31:03 2019][2765981.821914] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554921056/real 1554921056] req@ffff8b4eaaf61b00 x1630438570995456/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554921063 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 11:31:03 2019][2765981.849449] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Wed Apr 10 11:31:10 2019][2765988.860062] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554921063/real 1554921063] req@ffff8b4eaaf61b00 x1630438570995456/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554921070 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 11:31:10 2019][2765988.887571] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Wed Apr 10 11:31:17 2019][2765995.897218] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554921070/real 1554921070] req@ffff8b4eaaf61b00 x1630438570995456/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554921077 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 11:31:17 2019][2765995.924734] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [Wed Apr 10 11:31:31 2019][2766009.906508] Lustre: 118250:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554921084/real 1554921084] req@ffff8b6b84fa0600 x1630438570996032/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554921091 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 11:31:31 2019][2766009.934017] Lustre: 118250:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Wed Apr 10 11:31:52 2019][2766030.934954] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554921105/real 1554921105] req@ffff8b4eaaf61b00 x1630438570995456/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554921112 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 11:31:52 2019][2766030.962511] Lustre: 117551:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [Wed Apr 10 11:32:27 2019][2766065.946694] Lustre: 118250:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554921140/real 1554921140] req@ffff8b6b84fa0600 x1630438570996032/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554921147 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 11:32:27 2019][2766065.974212] Lustre: 118250:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [Wed Apr 10 11:33:31 2019][2766130.170522] LustreError: 118250:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.114.5@o2ib4) returned error from glimpse AST (req@ffff8b6b84fa0600 x1630438570996032 status -107 rc -107), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b8ccb250240/0x53d0f21439ccf9c3 lrc: 4/0,0 mode: PW/PW res: [0x2c00104c6:0xf2d3:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x40200000000000 nid: 10.9.114.5@o2ib4 remote: 0x7914cd393fc3833f expref: 1596 pid: 118667 timeout: 0 lvb_type: 0 [Wed Apr 10 11:33:31 2019][2766130.170529] LustreError: 138-a: fir-MDT0002: A client on nid 10.9.114.5@o2ib4 was evicted due to a lock glimpse callback time out: rc -107 [Wed Apr 10 11:33:31 2019][2766130.170549] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1168s: evicting client at 10.9.114.5@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8b6929f70480/0x53d0f21439cc0819 lrc: 4/0,0 mode: PW/PW res: [0x2c0010604:0xa69b:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x40200000000000 nid: 10.9.114.5@o2ib4 remote: 0x7914cd393fc37722 expref: 1597 pid: 118271 timeout: 0 lvb_type: 0 [Wed Apr 10 11:33:31 2019][2766130.264082] LustreError: 118250:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages [Wed Apr 10 11:33:32 2019][2766130.446426] Lustre: MGS: haven't heard from client 8dbd0a5f-b7c9-5b04-e651-811031c31d57 (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8cc75b1800, cur 1554921212 expire 1554921062 last 1554920985 [Wed Apr 10 11:33:32 2019][2766130.467628] Lustre: Skipped 2 previous similar messages [Wed Apr 10 11:33:34 2019][2766132.417109] Lustre: fir-MDT0000: haven't heard from client 6d1dbc37-4cd8-8085-186b-d10709d6a0dc (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b79c674e800, cur 1554921214 expire 1554921064 last 1554920987 [Wed Apr 10 11:35:57 2019][2766276.162669] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.11@o2ib4) [Wed Apr 10 11:35:57 2019][2766276.170174] Lustre: Skipped 412 previous similar messages [Wed Apr 10 11:36:13 2019][2766292.090569] Lustre: fir-MDT0002: Client 01f19ed0-770e-84b4-7586-12f4195b7925 (at 10.9.108.34@o2ib4) reconnecting [Wed Apr 10 11:36:13 2019][2766292.100926] Lustre: Skipped 409 previous similar messages [Wed Apr 10 11:45:17 2019][2766835.432020] Lustre: fir-MDT0000: haven't heard from client 558f6111-251f-9917-b7f1-a1e757b9c3a9 (at 10.9.101.37@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b888f217c00, cur 1554921917 expire 1554921767 last 1554921690 [Wed Apr 10 11:45:40 2019][2766859.265542] Lustre: 118271:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 11:45:40 2019][2766859.265542] req@ffff8b6bbdf5a100 x1628649449843280/t146066804493(0) o36->70db6507-827a-2e40-1c44-099543d1512c@10.9.108.33@o2ib4:15/0 lens 544/2888 e 1 to 0 dl 1554921945 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:45:40 2019][2766859.295662] Lustre: 118271:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Apr 10 11:45:46 2019][2766865.277641] Lustre: 118224:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-1), not sending early reply [Wed Apr 10 11:45:46 2019][2766865.277641] req@ffff8b586222b300 x1629298194064272/t0(0) o101->ccbe839d-1880-0660-8c5e-d6ce0f68a97f@10.9.108.31@o2ib4:21/0 lens 576/3264 e 1 to 0 dl 1554921951 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 11:46:01 2019][2766880.187542] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.34@o2ib4) [Wed Apr 10 11:46:01 2019][2766880.195037] Lustre: Skipped 406 previous similar messages [Wed Apr 10 11:46:13 2019][2766892.258047] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Wed Apr 10 11:46:13 2019][2766892.268400] Lustre: Skipped 404 previous similar messages [Wed Apr 10 11:46:55 2019][2766934.235101] LustreError: 118435:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554921925, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b58798bca40/0x53d0f2145019425d lrc: 3/1,0 mode: --/PR res: [0x2c001065c:0xc73:0x0].0x0 bits 0x13/0x8 rrc: 12 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118435 timeout: 0 lvb_type: 0 [Wed Apr 10 11:46:55 2019][2766934.274863] LustreError: 118435:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 11:48:46 2019][2767044.701439] LNet: Service thread pid 118435 was inactive for 200.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 11:48:46 2019][2767044.718654] LNet: Skipped 2 previous similar messages [Wed Apr 10 11:48:46 2019][2767044.723886] Pid: 118435, comm: mdt00_078 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:48:46 2019][2767044.733918] Call Trace: [Wed Apr 10 11:48:46 2019][2767044.736565] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:48:46 2019][2767044.743702] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:48:46 2019][2767044.751069] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 11:48:46 2019][2767044.758072] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 11:48:46 2019][2767044.765249] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 11:48:46 2019][2767044.771904] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 11:48:46 2019][2767044.779090] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 11:48:46 2019][2767044.785834] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 11:48:46 2019][2767044.792511] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 11:48:46 2019][2767044.799455] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 11:48:46 2019][2767044.806752] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 11:48:46 2019][2767044.813096] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:48:46 2019][2767044.820229] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:48:46 2019][2767044.828127] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:48:46 2019][2767044.834633] [] kthread+0xd1/0xe0 [Wed Apr 10 11:48:46 2019][2767044.839724] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:48:46 2019][2767044.846370] [] 0xffffffffffffffff [Wed Apr 10 11:48:46 2019][2767044.851579] LustreError: dumping log to /tmp/lustre-log.1554922126.118435 [Wed Apr 10 11:48:47 2019][2767046.116065] LNet: Service thread pid 118371 was inactive for 201.87s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 11:48:47 2019][2767046.133259] Pid: 118371, comm: mdt01_093 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 11:48:47 2019][2767046.143258] Call Trace: [Wed Apr 10 11:48:47 2019][2767046.145898] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 11:48:47 2019][2767046.153040] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 11:48:47 2019][2767046.160414] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 11:48:47 2019][2767046.167339] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 11:48:47 2019][2767046.173993] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 11:48:47 2019][2767046.180231] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 11:48:47 2019][2767046.186996] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 11:48:47 2019][2767046.192889] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 11:48:47 2019][2767046.200040] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 11:48:47 2019][2767046.207973] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 11:48:47 2019][2767046.214491] [] kthread+0xd1/0xe0 [Wed Apr 10 11:48:47 2019][2767046.219584] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 11:48:47 2019][2767046.226265] [] 0xffffffffffffffff [Wed Apr 10 11:51:27 2019][2767206.194348] LustreError: 119006:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128f5:0x92:0x0] [Wed Apr 10 11:56:01 2019][2767480.401342] Lustre: fir-MDT0002: Connection restored to bcead966-3991-3be5-a9ca-eba502e01069 (at 10.9.108.31@o2ib4) [Wed Apr 10 11:56:02 2019][2767480.411958] Lustre: Skipped 439 previous similar messages [Wed Apr 10 11:56:16 2019][2767494.419708] Lustre: fir-MDT0002: Client f53d399b-a6b5-36d0-034f-8c8c0efbb65f (at 10.9.106.11@o2ib4) reconnecting [Wed Apr 10 11:56:16 2019][2767494.430062] Lustre: Skipped 438 previous similar messages [Wed Apr 10 12:06:04 2019][2768082.551833] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.11@o2ib4) [Wed Apr 10 12:06:04 2019][2768082.559328] Lustre: Skipped 438 previous similar messages [Wed Apr 10 12:06:18 2019][2768096.723211] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Wed Apr 10 12:06:18 2019][2768096.733561] Lustre: Skipped 441 previous similar messages [Wed Apr 10 12:10:48 2019][2768367.018944] LustreError: 119135:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001120e:0x1d03:0x0] [Wed Apr 10 12:11:47 2019][2768425.465612] Lustre: fir-MDT0000: haven't heard from client 4be11fe3-b6ee-1f5f-0a23-14b3c54d5e8d (at 10.9.106.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b886830c400, cur 1554923507 expire 1554923357 last 1554923280 [Wed Apr 10 12:11:47 2019][2768425.487574] Lustre: Skipped 2 previous similar messages [Wed Apr 10 12:14:33 2019][2768591.468944] Lustre: fir-MDT0002: haven't heard from client 2e0c4648-6e21-c09c-2281-ea766335a2a8 (at 10.9.101.35@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7634af0c00, cur 1554923673 expire 1554923523 last 1554923446 [Wed Apr 10 12:14:33 2019][2768591.490998] Lustre: Skipped 2 previous similar messages [Wed Apr 10 12:16:07 2019][2768685.840635] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 12:16:07 2019][2768685.848126] Lustre: Skipped 444 previous similar messages [Wed Apr 10 12:16:18 2019][2768697.415877] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 12:16:18 2019][2768697.426236] Lustre: Skipped 436 previous similar messages [Wed Apr 10 12:16:37 2019][2768715.596734] LustreError: 119007:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001124a:0x1022:0x0] [Wed Apr 10 12:17:53 2019][2768791.516758] LustreError: 118512:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b8cc273d700 x1628639100114672/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:29/0 lens 448/440 e 0 to 0 dl 1554923879 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 12:21:07 2019][2768985.551308] LustreError: 118443:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010496:0xa58:0x0] [Wed Apr 10 12:21:38 2019][2769016.766714] Lustre: 118558:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 12:21:38 2019][2769016.766714] req@ffff8b63d7201200 x1629299613286016/t146080203332(0) o36->58899832-571e-4a18-664b-6f5c1308e27e@10.8.18.18@o2ib6:13/0 lens 536/2888 e 1 to 0 dl 1554924103 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 12:21:42 2019][2769021.286801] Lustre: 118431:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 12:21:42 2019][2769021.286801] req@ffff8b745feeaa00 x1629297378817584/t146080280468(0) o36->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:17/0 lens 504/2888 e 1 to 0 dl 1554924107 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 12:21:42 2019][2769021.316749] Lustre: 118431:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 12:22:53 2019][2769091.740257] LustreError: 118455:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554924083, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b64b42f6300/0x53d0f214722b4f99 lrc: 3/1,0 mode: --/PR res: [0x2c001124e:0xb94:0x0].0x0 bits 0x13/0x8 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118455 timeout: 0 lvb_type: 0 [Wed Apr 10 12:22:53 2019][2769091.779930] LustreError: 118455:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 12:22:57 2019][2769096.260350] LustreError: 118218:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554924087, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b77faa60900/0x53d0f2147247a7b3 lrc: 3/1,0 mode: --/PR res: [0x2c001124a:0x114c:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118218 timeout: 0 lvb_type: 0 [Wed Apr 10 12:22:57 2019][2769096.300117] LustreError: 118218:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 12:24:43 2019][2769201.802537] LNet: Service thread pid 118206 was inactive for 200.06s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 12:24:43 2019][2769201.819731] Pid: 118206, comm: mdt00_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 12:24:43 2019][2769201.829735] Call Trace: [Wed Apr 10 12:24:43 2019][2769201.832397] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 12:24:43 2019][2769201.839525] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 12:24:43 2019][2769201.846907] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 12:24:43 2019][2769201.853832] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 12:24:43 2019][2769201.860487] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 12:24:43 2019][2769201.866709] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 12:24:43 2019][2769201.873456] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 12:24:43 2019][2769201.879338] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 12:24:43 2019][2769201.886471] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 12:24:43 2019][2769201.894376] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 12:24:43 2019][2769201.900901] [] kthread+0xd1/0xe0 [Wed Apr 10 12:24:43 2019][2769201.905997] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 12:24:43 2019][2769201.912654] [] 0xffffffffffffffff [Wed Apr 10 12:24:43 2019][2769201.917854] LustreError: dumping log to /tmp/lustre-log.1554924283.118206 [Wed Apr 10 12:24:44 2019][2769203.252461] LNet: Service thread pid 118455 was inactive for 201.50s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 12:24:44 2019][2769203.269678] Pid: 118455, comm: mdt00_084 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 12:24:44 2019][2769203.279677] Call Trace: [Wed Apr 10 12:24:44 2019][2769203.282318] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 12:24:44 2019][2769203.289437] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 12:24:44 2019][2769203.296848] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 12:24:44 2019][2769203.303851] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 12:24:44 2019][2769203.311027] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 12:24:44 2019][2769203.317683] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 12:24:44 2019][2769203.324859] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 12:24:44 2019][2769203.331600] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 12:24:44 2019][2769203.338257] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 12:24:44 2019][2769203.345216] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 12:24:44 2019][2769203.352515] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 12:24:44 2019][2769203.358868] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 12:24:44 2019][2769203.365993] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 12:24:44 2019][2769203.373919] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 12:24:44 2019][2769203.380440] [] kthread+0xd1/0xe0 [Wed Apr 10 12:24:44 2019][2769203.385525] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 12:24:44 2019][2769203.392201] [] 0xffffffffffffffff [Wed Apr 10 12:24:48 2019][2769206.922639] LNet: Service thread pid 118218 was inactive for 200.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 12:24:48 2019][2769206.939833] Pid: 118218, comm: mdt02_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 12:24:48 2019][2769206.949836] Call Trace: [Wed Apr 10 12:24:48 2019][2769206.952500] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769206.959661] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769206.967070] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 12:24:48 2019][2769206.974097] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 12:24:48 2019][2769206.981294] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 12:24:48 2019][2769206.987956] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 12:24:48 2019][2769206.995139] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 12:24:48 2019][2769207.001883] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 12:24:48 2019][2769207.008539] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.015500] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.022788] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.029149] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.036273] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.044187] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.050695] [] kthread+0xd1/0xe0 [Wed Apr 10 12:24:48 2019][2769207.055783] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 12:24:48 2019][2769207.062443] [] 0xffffffffffffffff [Wed Apr 10 12:24:48 2019][2769207.067658] LustreError: dumping log to /tmp/lustre-log.1554924288.118218 [Wed Apr 10 12:24:48 2019][2769207.107050] Pid: 118392, comm: mdt02_050 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 12:24:48 2019][2769207.117072] Call Trace: [Wed Apr 10 12:24:48 2019][2769207.119712] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.126835] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.134217] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 12:24:48 2019][2769207.141169] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 12:24:48 2019][2769207.147833] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 12:24:48 2019][2769207.154058] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 12:24:48 2019][2769207.160825] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 12:24:48 2019][2769207.166700] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.173853] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.181759] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 12:24:48 2019][2769207.188306] [] kthread+0xd1/0xe0 [Wed Apr 10 12:24:48 2019][2769207.193403] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 12:24:48 2019][2769207.200052] [] 0xffffffffffffffff [Wed Apr 10 12:26:08 2019][2769286.848289] Lustre: fir-MDT0002: Connection restored to bcead966-3991-3be5-a9ca-eba502e01069 (at 10.9.108.31@o2ib4) [Wed Apr 10 12:26:08 2019][2769286.858901] Lustre: Skipped 480 previous similar messages [Wed Apr 10 12:26:22 2019][2769300.642772] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Wed Apr 10 12:26:22 2019][2769300.652957] Lustre: Skipped 481 previous similar messages [Wed Apr 10 12:28:34 2019][2769432.463261] Lustre: 118547:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 12:28:34 2019][2769432.463261] req@ffff8b559cf12700 x1630376962913408/t0(0) o101->45f27c78-9a7f-da99-6869-a1357a032459@10.8.27.8@o2ib6:8/0 lens 576/3264 e 0 to 0 dl 1554924518 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 12:28:34 2019][2769432.492344] Lustre: 118547:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 12:28:54 2019][2769452.886888] LustreError: 118443:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001078f:0x10e1:0x0] [Wed Apr 10 12:29:38 2019][2769497.405579] LustreError: 118471:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554924488, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b5cc62672c0/0x53d0f2147961a683 lrc: 3/1,0 mode: --/PR res: [0x2c001119d:0x63a:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118471 timeout: 0 lvb_type: 0 [Wed Apr 10 12:29:38 2019][2769497.445256] LustreError: 118471:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 12:31:00 2019][2769578.512265] Lustre: 118517:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 12:31:00 2019][2769578.512265] req@ffff8b74193c8900 x1629300767324000/t146085043696(0) o36->26c07b92-3e88-a582-abe6-28f16a4cf62b@10.9.108.26@o2ib4:5/0 lens 544/2888 e 0 to 0 dl 1554924665 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 12:31:29 2019][2769607.826871] LNet: Service thread pid 118471 was inactive for 200.41s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 12:31:29 2019][2769607.844063] LNet: Skipped 1 previous similar message [Wed Apr 10 12:31:29 2019][2769607.849215] Pid: 118471, comm: mdt00_088 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 12:31:29 2019][2769607.859211] Call Trace: [Wed Apr 10 12:31:29 2019][2769607.861847] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 12:31:29 2019][2769607.868980] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 12:31:29 2019][2769607.876362] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 12:31:29 2019][2769607.883365] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 12:31:29 2019][2769607.890539] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 12:31:29 2019][2769607.897215] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 12:31:29 2019][2769607.904400] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 12:31:29 2019][2769607.911157] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 12:31:29 2019][2769607.917814] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 12:31:29 2019][2769607.924748] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 12:31:29 2019][2769607.932030] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 12:31:29 2019][2769607.938364] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 12:31:29 2019][2769607.945479] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 12:31:29 2019][2769607.953368] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 12:31:29 2019][2769607.959870] [] kthread+0xd1/0xe0 [Wed Apr 10 12:31:29 2019][2769607.964961] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 12:31:29 2019][2769607.971627] [] 0xffffffffffffffff [Wed Apr 10 12:31:29 2019][2769607.976822] LustreError: dumping log to /tmp/lustre-log.1554924689.118471 [Wed Apr 10 12:31:30 2019][2769609.058431] Pid: 118342, comm: mdt01_079 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 12:31:30 2019][2769609.068429] Call Trace: [Wed Apr 10 12:31:30 2019][2769609.071066] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 12:31:30 2019][2769609.078188] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 12:31:30 2019][2769609.085553] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 12:31:30 2019][2769609.092479] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 12:31:30 2019][2769609.099135] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 12:31:30 2019][2769609.105359] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 12:31:30 2019][2769609.112099] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 12:31:30 2019][2769609.117976] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 12:31:30 2019][2769609.125122] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 12:31:30 2019][2769609.133025] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 12:31:30 2019][2769609.139532] [] kthread+0xd1/0xe0 [Wed Apr 10 12:31:30 2019][2769609.144617] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 12:31:30 2019][2769609.151275] [] 0xffffffffffffffff [Wed Apr 10 12:32:05 2019][2769643.975635] LustreError: 118303:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554924635, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b8905b2ca40/0x53d0f2147cf5b3ac lrc: 3/1,0 mode: --/PR res: [0x2c0011195:0x14801:0x0].0x0 bits 0x13/0x8 rrc: 19 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118303 timeout: 0 lvb_type: 0 [Wed Apr 10 12:32:05 2019][2769644.015588] LustreError: 118303:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 12:33:55 2019][2769754.261935] LNet: Service thread pid 118252 was inactive for 200.28s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 12:33:55 2019][2769754.279145] LNet: Skipped 1 previous similar message [Wed Apr 10 12:33:55 2019][2769754.284294] Pid: 118252, comm: mdt02_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 12:33:55 2019][2769754.294322] Call Trace: [Wed Apr 10 12:33:55 2019][2769754.296965] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 12:33:55 2019][2769754.304083] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 12:33:55 2019][2769754.311482] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 12:33:55 2019][2769754.318417] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 12:33:55 2019][2769754.325075] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 12:33:55 2019][2769754.331298] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 12:33:55 2019][2769754.338057] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 12:33:55 2019][2769754.343943] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 12:33:55 2019][2769754.351077] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 12:33:55 2019][2769754.358987] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 12:33:55 2019][2769754.365503] [] kthread+0xd1/0xe0 [Wed Apr 10 12:33:55 2019][2769754.370594] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 12:33:55 2019][2769754.377240] [] 0xffffffffffffffff [Wed Apr 10 12:33:55 2019][2769754.382433] LustreError: dumping log to /tmp/lustre-log.1554924835.118252 [Wed Apr 10 12:33:56 2019][2769755.046014] Pid: 118303, comm: mdt00_056 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 12:33:56 2019][2769755.056017] Call Trace: [Wed Apr 10 12:33:56 2019][2769755.058662] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 12:33:56 2019][2769755.065780] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 12:33:56 2019][2769755.073166] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 12:33:56 2019][2769755.080178] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 12:33:56 2019][2769755.087367] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 12:33:56 2019][2769755.094025] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 12:33:56 2019][2769755.101217] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 12:33:56 2019][2769755.107962] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 12:33:56 2019][2769755.114631] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 12:33:56 2019][2769755.121578] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 12:33:56 2019][2769755.128857] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 12:33:56 2019][2769755.135201] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 12:33:56 2019][2769755.142318] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 12:33:56 2019][2769755.150212] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 12:33:56 2019][2769755.156720] [] kthread+0xd1/0xe0 [Wed Apr 10 12:33:56 2019][2769755.161812] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 12:33:56 2019][2769755.168465] [] 0xffffffffffffffff [Wed Apr 10 12:36:10 2019][2769888.839993] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Wed Apr 10 12:36:10 2019][2769888.850450] Lustre: Skipped 577 previous similar messages [Wed Apr 10 12:36:25 2019][2769903.784941] Lustre: fir-MDT0002: Client 45f27c78-9a7f-da99-6869-a1357a032459 (at 10.8.27.8@o2ib6) reconnecting [Wed Apr 10 12:36:25 2019][2769903.795119] Lustre: Skipped 579 previous similar messages [Wed Apr 10 12:37:54 2019][2769992.497710] Lustre: fir-MDT0000: haven't heard from client 55aafc93-9dff-fa25-67e4-1bdcf1bf214f (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5f527be800, cur 1554925074 expire 1554924924 last 1554924847 [Wed Apr 10 12:37:54 2019][2769992.519506] Lustre: Skipped 2 previous similar messages [Wed Apr 10 12:40:45 2019][2770163.501268] Lustre: fir-MDT0002: haven't heard from client 55aafc93-9dff-fa25-67e4-1bdcf1bf214f (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6f4f264800, cur 1554925245 expire 1554925095 last 1554925018 [Wed Apr 10 12:40:45 2019][2770163.523056] Lustre: Skipped 1 previous similar message [Wed Apr 10 12:42:01 2019][2770239.502864] Lustre: fir-MDT0002: haven't heard from client 4e7a41ff-599c-6fd3-1588-d7b9b8fbf455 (at 10.9.114.5@o2ib4) in 193 seconds. I think it's dead, and I am evicting it. exp ffff8b595c3b6400, cur 1554925321 expire 1554925171 last 1554925128 [Wed Apr 10 12:42:30 2019][2770268.504422] Lustre: fir-MDT0000: haven't heard from client 4e7a41ff-599c-6fd3-1588-d7b9b8fbf455 (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b77ef761c00, cur 1554925350 expire 1554925200 last 1554925123 [Wed Apr 10 12:42:35 2019][2770273.516242] Lustre: MGS: haven't heard from client 5112c2f6-bd95-a4fd-17ab-490d4d44758d (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5cb55eec00, cur 1554925355 expire 1554925205 last 1554925128 [Wed Apr 10 12:46:14 2019][2770492.690889] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.34@o2ib4) [Wed Apr 10 12:46:14 2019][2770492.698380] Lustre: Skipped 611 previous similar messages [Wed Apr 10 12:46:26 2019][2770505.099276] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Wed Apr 10 12:46:26 2019][2770505.109625] Lustre: Skipped 602 previous similar messages [Wed Apr 10 12:48:42 2019][2770640.511137] Lustre: fir-MDT0000: haven't heard from client 618e27a5-1163-e0cc-db2b-5b7bd2c4fd87 (at 10.9.101.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8be46da800, cur 1554925722 expire 1554925572 last 1554925495 [Wed Apr 10 12:49:58 2019][2770716.512737] Lustre: fir-MDT0000: haven't heard from client 2cfc41a5-db0c-965a-d142-7bbd5278a4b1 (at 10.9.104.38@o2ib4) in 219 seconds. I think it's dead, and I am evicting it. exp ffff8b4de6990c00, cur 1554925798 expire 1554925648 last 1554925579 [Wed Apr 10 12:49:58 2019][2770716.534715] Lustre: Skipped 2 previous similar messages [Wed Apr 10 12:56:14 2019][2771093.228832] Lustre: fir-MDT0002: Connection restored to bcead966-3991-3be5-a9ca-eba502e01069 (at 10.9.108.31@o2ib4) [Wed Apr 10 12:56:14 2019][2771093.239486] Lustre: Skipped 605 previous similar messages [Wed Apr 10 12:56:28 2019][2771107.052149] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Wed Apr 10 12:56:28 2019][2771107.062331] Lustre: Skipped 604 previous similar messages [Wed Apr 10 13:06:14 2019][2771693.339966] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 13:06:14 2019][2771693.347453] Lustre: Skipped 601 previous similar messages [Wed Apr 10 13:06:32 2019][2771710.890078] Lustre: fir-MDT0002: Client 01f19ed0-770e-84b4-7586-12f4195b7925 (at 10.9.108.34@o2ib4) reconnecting [Wed Apr 10 13:06:32 2019][2771710.900426] Lustre: Skipped 603 previous similar messages [Wed Apr 10 13:16:20 2019][2772298.990203] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.34@o2ib4) [Wed Apr 10 13:16:20 2019][2772298.997704] Lustre: Skipped 609 previous similar messages [Wed Apr 10 13:16:32 2019][2772311.493926] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Wed Apr 10 13:16:32 2019][2772311.504273] Lustre: Skipped 601 previous similar messages [Wed Apr 10 13:26:21 2019][2772899.632872] Lustre: fir-MDT0002: Connection restored to bcead966-3991-3be5-a9ca-eba502e01069 (at 10.9.108.31@o2ib4) [Wed Apr 10 13:26:21 2019][2772899.643482] Lustre: Skipped 604 previous similar messages [Wed Apr 10 13:26:34 2019][2772913.367595] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Wed Apr 10 13:26:34 2019][2772913.377768] Lustre: Skipped 605 previous similar messages [Wed Apr 10 13:27:53 2019][2772991.684421] Lustre: 118389:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554928066/real 1554928066] req@ffff8b621c2e8300 x1630438691561952/t0(0) o104->fir-MDT0002@10.8.10.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554928073 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 13:27:53 2019][2772991.711931] Lustre: 118389:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 28 previous similar messages [Wed Apr 10 13:28:01 2019][2772999.718591] Lustre: 117534:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 13:28:01 2019][2772999.718591] req@ffff8b7551f83c00 x1628595390934128/t0(0) o101->20224108-ace5-594c-deee-f7b4391d6b01@10.8.21.16@o2ib6:6/0 lens 1784/3288 e 1 to 0 dl 1554928086 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 13:28:01 2019][2772999.747949] Lustre: 117534:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 13:28:07 2019][2773005.721717] Lustre: 118389:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554928080/real 1554928080] req@ffff8b621c2e8300 x1630438691561952/t0(0) o104->fir-MDT0002@10.8.10.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554928087 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 13:28:07 2019][2773005.749244] Lustre: 118389:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Wed Apr 10 13:28:28 2019][2773026.759162] Lustre: 118389:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554928101/real 1554928101] req@ffff8b621c2e8300 x1630438691561952/t0(0) o104->fir-MDT0002@10.8.10.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554928108 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 13:28:28 2019][2773026.786962] Lustre: 118389:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Wed Apr 10 13:29:03 2019][2773061.797896] Lustre: 118389:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554928136/real 1554928136] req@ffff8b621c2e8300 x1630438691561952/t0(0) o104->fir-MDT0002@10.8.10.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554928143 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 13:29:03 2019][2773061.825408] Lustre: 118389:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Wed Apr 10 13:29:16 2019][2773074.699164] LustreError: 118408:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554928066, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b7968a6de80/0x53d0f214b1e0fab2 lrc: 3/1,0 mode: --/PR res: [0x2c001191f:0x6e9a:0x0].0x0 bits 0x13/0x0 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118408 timeout: 0 lvb_type: 0 [Wed Apr 10 13:30:12 2019][2773130.562337] Lustre: fir-MDT0000: haven't heard from client 766c24dc-05fc-2c20-7902-bb94de21535e (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b870d31dc00, cur 1554928212 expire 1554928062 last 1554927985 [Wed Apr 10 13:30:12 2019][2773130.584229] Lustre: Skipped 2 previous similar messages [Wed Apr 10 13:33:13 2019][2773311.566149] Lustre: fir-MDT0002: haven't heard from client 80c07668-3931-2a59-f84a-8b9dac6db9e5 (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b764628a000, cur 1554928393 expire 1554928243 last 1554928166 [Wed Apr 10 13:33:13 2019][2773311.588038] Lustre: Skipped 2 previous similar messages [Wed Apr 10 13:34:17 2019][2773376.210420] Lustre: 118266:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 13:34:17 2019][2773376.210420] req@ffff8b70f028d400 x1628648740370080/t146097660545(0) o36->837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579@10.9.104.22@o2ib4:22/0 lens 544/2888 e 1 to 0 dl 1554928462 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 13:34:17 2019][2773376.240557] Lustre: 118266:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 13:34:27 2019][2773385.968615] Lustre: 118782:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 13:34:27 2019][2773385.968615] req@ffff8b8a2f700c00 x1629293148904848/t0(0) o101->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:2/0 lens 576/3264 e 0 to 0 dl 1554928472 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 13:35:32 2019][2773450.916943] LustreError: 118463:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554928442, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b51d3371b00/0x53d0f214b829c546 lrc: 3/1,0 mode: --/PR res: [0x2c00107bd:0x7ad:0x0].0x0 bits 0x13/0x8 rrc: 11 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118463 timeout: 0 lvb_type: 0 [Wed Apr 10 13:35:32 2019][2773450.956708] LustreError: 118463:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 13:36:21 2019][2773500.307523] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.8@o2ib6) [Wed Apr 10 13:36:21 2019][2773500.314839] Lustre: Skipped 628 previous similar messages [Wed Apr 10 13:36:37 2019][2773516.202349] Lustre: fir-MDT0002: Client 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4) reconnecting [Wed Apr 10 13:36:37 2019][2773516.212706] Lustre: Skipped 624 previous similar messages [Wed Apr 10 13:37:22 2019][2773561.060284] LNet: Service thread pid 118258 was inactive for 200.14s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 13:37:22 2019][2773561.077497] LNet: Skipped 1 previous similar message [Wed Apr 10 13:37:22 2019][2773561.082641] Pid: 118258, comm: mdt02_028 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 13:37:22 2019][2773561.092641] Call Trace: [Wed Apr 10 13:37:22 2019][2773561.095287] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 13:37:22 2019][2773561.102424] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 13:37:22 2019][2773561.109790] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 13:37:22 2019][2773561.116715] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 13:37:22 2019][2773561.123369] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 13:37:22 2019][2773561.129594] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 13:37:22 2019][2773561.136336] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 13:37:22 2019][2773561.142228] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 13:37:22 2019][2773561.149355] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 13:37:22 2019][2773561.157248] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 13:37:22 2019][2773561.163758] [] kthread+0xd1/0xe0 [Wed Apr 10 13:37:22 2019][2773561.168859] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 13:37:22 2019][2773561.175511] [] 0xffffffffffffffff [Wed Apr 10 13:37:22 2019][2773561.180702] LustreError: dumping log to /tmp/lustre-log.1554928642.118258 [Wed Apr 10 13:37:23 2019][2773562.513047] LNet: Service thread pid 118463 was inactive for 201.59s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 13:37:23 2019][2773562.530241] Pid: 118463, comm: mdt03_047 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 13:37:23 2019][2773562.540241] Call Trace: [Wed Apr 10 13:37:23 2019][2773562.542877] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 13:37:23 2019][2773562.549999] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 13:37:23 2019][2773562.557367] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 13:37:24 2019][2773562.564384] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 13:37:24 2019][2773562.571562] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 13:37:24 2019][2773562.578219] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 13:37:24 2019][2773562.585393] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 13:37:24 2019][2773562.592136] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 13:37:24 2019][2773562.598793] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 13:37:24 2019][2773562.605735] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 13:37:24 2019][2773562.613015] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 13:37:24 2019][2773562.619374] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 13:37:24 2019][2773562.626494] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 13:37:24 2019][2773562.634394] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 13:37:24 2019][2773562.640908] [] kthread+0xd1/0xe0 [Wed Apr 10 13:37:24 2019][2773562.646016] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 13:37:24 2019][2773562.652666] [] 0xffffffffffffffff [Wed Apr 10 13:40:05 2019][2773723.589749] Lustre: 118424:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554928798/real 1554928798] req@ffff8b8b992ce600 x1630438702585584/t0(0) o106->fir-MDT0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554928805 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 13:40:05 2019][2773723.617283] Lustre: 118424:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [Wed Apr 10 13:40:23 2019][2773741.651131] Lustre: 118373:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 13:40:23 2019][2773741.651131] req@ffff8b8bd537bc00 x1629928156629616/t0(0) o101->d688f6d7-017e-8817-dbcd-6b016c3633d8@10.9.0.61@o2ib4:28/0 lens 480/568 e 0 to 0 dl 1554928828 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 13:42:18 2019][2773856.634565] Lustre: 118424:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554928931/real 1554928931] req@ffff8b8b992ce600 x1630438702585584/t0(0) o106->fir-MDT0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554928938 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 13:42:18 2019][2773856.662098] Lustre: 118424:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages [Wed Apr 10 13:42:39 2019][2773877.675145] LustreError: 118424:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from glimpse AST (req@ffff8b8b992ce600 x1630438702585584 status -107 rc -107), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8b57d820f080/0x53d0f214b99059cc lrc: 4/0,0 mode: PW/PW res: [0x20000f949:0x315:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x40200000000000 nid: 10.8.27.23@o2ib6 remote: 0x2c978d64b4c3c89c expref: 347 pid: 118280 timeout: 0 lvb_type: 0 [Wed Apr 10 13:42:39 2019][2773877.718040] LustreError: 138-a: fir-MDT0000: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 [Wed Apr 10 13:42:39 2019][2773877.730642] LustreError: Skipped 2 previous similar messages [Wed Apr 10 13:42:39 2019][2773877.736516] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 448s: evicting client at 10.8.27.23@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8b57d820f080/0x53d0f214b99059cc lrc: 4/0,0 mode: PW/PW res: [0x20000f949:0x315:0x0].0x0 bits 0x40/0x0 rrc: 7 type: IBT flags: 0x40200000000000 nid: 10.8.27.23@o2ib6 remote: 0x2c978d64b4c3c89c expref: 348 pid: 118280 timeout: 0 lvb_type: 0 [Wed Apr 10 13:42:39 2019][2773877.774184] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Apr 10 13:43:18 2019][2773916.596423] Lustre: MGS: haven't heard from client 5a0b2157-78b0-d197-bec8-f08e8864a03c (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b873624cc00, cur 1554928998 expire 1554928848 last 1554928771 [Wed Apr 10 13:43:18 2019][2773916.617613] Lustre: Skipped 2 previous similar messages [Wed Apr 10 13:46:26 2019][2774105.279990] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.34@o2ib4) [Wed Apr 10 13:46:26 2019][2774105.287479] Lustre: Skipped 659 previous similar messages [Wed Apr 10 13:46:37 2019][2774116.299364] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 13:46:37 2019][2774116.309717] Lustre: Skipped 655 previous similar messages [Wed Apr 10 13:47:46 2019][2774184.584549] Lustre: fir-MDT0000: haven't heard from client 98af2bbe-c574-a824-5582-485a32691920 (at 10.9.101.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8bfa7b5000, cur 1554929266 expire 1554929116 last 1554929039 [Wed Apr 10 13:47:46 2019][2774184.606512] Lustre: Skipped 1 previous similar message [Wed Apr 10 13:50:33 2019][2774351.588075] Lustre: fir-MDT0002: haven't heard from client c7a325a4-90c0-0ef9-2b1d-f81ba35ea8f1 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7566211400, cur 1554929433 expire 1554929283 last 1554929206 [Wed Apr 10 13:50:33 2019][2774351.610007] Lustre: Skipped 2 previous similar messages [Wed Apr 10 13:56:27 2019][2774705.689468] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.22@o2ib4) [Wed Apr 10 13:56:27 2019][2774705.696961] Lustre: Skipped 661 previous similar messages [Wed Apr 10 13:56:41 2019][2774719.661511] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Wed Apr 10 13:56:41 2019][2774719.671687] Lustre: Skipped 652 previous similar messages [Wed Apr 10 13:58:44 2019][2774842.598611] Lustre: fir-MDT0000: haven't heard from client a2be9cb9-225a-12bb-3d5f-c18b08c767d0 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6205261000, cur 1554929924 expire 1554929774 last 1554929697 [Wed Apr 10 13:58:44 2019][2774842.620511] Lustre: Skipped 5 previous similar messages [Wed Apr 10 14:01:54 2019][2775032.602524] Lustre: fir-MDT0000: haven't heard from client d40514bf-f5f1-614f-2e6c-056d786841f0 (at 10.8.24.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4ce877f800, cur 1554930114 expire 1554929964 last 1554929887 [Wed Apr 10 14:01:54 2019][2775032.624427] Lustre: Skipped 2 previous similar messages [Wed Apr 10 14:03:09 2019][2775107.972838] LustreError: 118774:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012932:0x1195:0x0] [Wed Apr 10 14:06:29 2019][2775307.761389] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Wed Apr 10 14:06:29 2019][2775307.771851] Lustre: Skipped 656 previous similar messages [Wed Apr 10 14:06:42 2019][2775320.901674] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Wed Apr 10 14:06:42 2019][2775320.912033] Lustre: Skipped 651 previous similar messages [Wed Apr 10 14:07:31 2019][2775369.609655] Lustre: fir-MDT0000: haven't heard from client 92910762-2800-6855-c3ae-478c6013cc25 (at 10.8.31.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b886830b000, cur 1554930451 expire 1554930301 last 1554930224 [Wed Apr 10 14:07:31 2019][2775369.631442] Lustre: Skipped 2 previous similar messages [Wed Apr 10 14:16:31 2019][2775910.017122] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 14:16:31 2019][2775910.024613] Lustre: Skipped 663 previous similar messages [Wed Apr 10 14:16:43 2019][2775921.621280] Lustre: fir-MDT0000: haven't heard from client 98179822-15bb-223e-365a-0f77834fe7fa (at 10.8.10.1@o2ib6) in 202 seconds. I think it's dead, and I am evicting it. exp ffff8b59620eec00, cur 1554931003 expire 1554930853 last 1554930801 [Wed Apr 10 14:16:43 2019][2775921.643083] Lustre: Skipped 14 previous similar messages [Wed Apr 10 14:16:43 2019][2775922.586349] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 14:16:43 2019][2775922.596722] Lustre: Skipped 650 previous similar messages [Wed Apr 10 14:26:32 2019][2776510.697624] Lustre: fir-MDT0002: Connection restored to (at 10.9.103.11@o2ib4) [Wed Apr 10 14:26:32 2019][2776510.705115] Lustre: Skipped 653 previous similar messages [Wed Apr 10 14:26:45 2019][2776524.024760] Lustre: fir-MDT0002: Client 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4) reconnecting [Wed Apr 10 14:26:45 2019][2776524.035143] Lustre: Skipped 650 previous similar messages [Wed Apr 10 14:27:10 2019][2776548.634352] Lustre: fir-MDT0000: haven't heard from client eacb59ca-647f-dd3d-4d0d-fe167b61d004 (at 10.8.9.2@o2ib6) in 172 seconds. I think it's dead, and I am evicting it. exp ffff8b52b42e8c00, cur 1554931630 expire 1554931480 last 1554931458 [Wed Apr 10 14:27:10 2019][2776548.656060] Lustre: Skipped 8 previous similar messages [Wed Apr 10 14:36:34 2019][2777113.052665] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.19@o2ib4) [Wed Apr 10 14:36:34 2019][2777113.060157] Lustre: Skipped 671 previous similar messages [Wed Apr 10 14:36:45 2019][2777124.096273] Lustre: fir-MDT0002: Client 7d8ca85d-8b80-6a23-8fa9-83dca7eb7196 (at 10.8.27.28@o2ib6) reconnecting [Wed Apr 10 14:36:45 2019][2777124.106544] Lustre: Skipped 652 previous similar messages [Wed Apr 10 14:39:39 2019][2777297.863026] LustreError: 119203:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5f0cbb2400 x1628639116164176/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:15/0 lens 448/440 e 0 to 0 dl 1554932385 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 14:46:38 2019][2777716.929734] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.8@o2ib6) [Wed Apr 10 14:46:38 2019][2777716.937048] Lustre: Skipped 654 previous similar messages [Wed Apr 10 14:46:50 2019][2777728.901889] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 14:46:50 2019][2777728.912237] Lustre: Skipped 649 previous similar messages [Wed Apr 10 14:48:20 2019][2777818.661664] Lustre: fir-MDT0002: haven't heard from client 04c36c72-b4ce-d99a-b97d-bb0d21c053dc (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b71c8f33c00, cur 1554932900 expire 1554932750 last 1554932673 [Wed Apr 10 14:48:20 2019][2777818.683539] Lustre: Skipped 11 previous similar messages [Wed Apr 10 14:48:57 2019][2777856.084392] Lustre: 118450:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 14:48:57 2019][2777856.084392] req@ffff8b71f2aa1500 x1628640596610224/t146113083489(0) o36->86cc2579-6592-d6be-ad47-10ff366dfbdf@10.8.8.19@o2ib6:2/0 lens 544/2888 e 1 to 0 dl 1554932942 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 14:50:12 2019][2777931.398958] LustreError: 118354:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554932922, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b6ce7fe86c0/0x53d0f214f9b60085 lrc: 3/1,0 mode: --/PR res: [0x2c0012129:0x2172:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118354 timeout: 0 lvb_type: 0 [Wed Apr 10 14:50:12 2019][2777931.438709] LustreError: 118354:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 14:51:52 2019][2778031.064041] Lustre: 118431:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 14:51:52 2019][2778031.064041] req@ffff8b71f2eb4200 x1628594726511200/t0(0) o101->267d8456-47e2-363d-5081-c23100dd5f9b@10.8.27.20@o2ib6:27/0 lens 576/3264 e 1 to 0 dl 1554933117 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 14:51:52 2019][2778031.093247] Lustre: 118431:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 14:52:02 2019][2778040.952247] Lustre: 118232:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 14:52:02 2019][2778040.952247] req@ffff8b6a773eec00 x1629295855865632/t146113863790(0) o36->e18a93c6-a11b-0b52-af65-1483de6beb82@10.8.8.18@o2ib6:7/0 lens 544/2888 e 0 to 0 dl 1554933127 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 14:52:03 2019][2778042.178263] LNet: Service thread pid 118354 was inactive for 200.77s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 14:52:03 2019][2778042.195467] Pid: 118354, comm: mdt01_087 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 14:52:03 2019][2778042.205472] Call Trace: [Wed Apr 10 14:52:03 2019][2778042.208110] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 14:52:03 2019][2778042.215217] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 14:52:03 2019][2778042.222585] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 14:52:03 2019][2778042.229589] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 14:52:03 2019][2778042.236755] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 14:52:03 2019][2778042.243419] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 14:52:03 2019][2778042.250586] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 14:52:03 2019][2778042.257330] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 14:52:03 2019][2778042.263975] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 14:52:03 2019][2778042.270908] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 14:52:03 2019][2778042.278189] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 14:52:03 2019][2778042.284526] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 14:52:03 2019][2778042.291656] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 14:52:03 2019][2778042.299553] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 14:52:03 2019][2778042.306056] [] kthread+0xd1/0xe0 [Wed Apr 10 14:52:03 2019][2778042.311142] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 14:52:03 2019][2778042.317790] [] 0xffffffffffffffff [Wed Apr 10 14:52:03 2019][2778042.322983] LustreError: dumping log to /tmp/lustre-log.1554933123.118354 [Wed Apr 10 14:52:05 2019][2778043.665522] LNet: Service thread pid 118577 was inactive for 202.26s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 14:52:05 2019][2778043.682752] Pid: 118577, comm: mdt02_101 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 14:52:05 2019][2778043.692771] Call Trace: [Wed Apr 10 14:52:05 2019][2778043.695405] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 14:52:05 2019][2778043.702515] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 14:52:05 2019][2778043.709911] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 14:52:05 2019][2778043.716842] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 14:52:05 2019][2778043.723503] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 14:52:05 2019][2778043.729727] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 14:52:05 2019][2778043.736474] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 14:52:05 2019][2778043.742362] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 14:52:05 2019][2778043.749485] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 14:52:05 2019][2778043.757381] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 14:52:05 2019][2778043.763882] [] kthread+0xd1/0xe0 [Wed Apr 10 14:52:05 2019][2778043.768990] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 14:52:05 2019][2778043.775643] [] 0xffffffffffffffff [Wed Apr 10 14:52:15 2019][2778053.984518] Lustre: 117467:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 14:52:15 2019][2778053.984518] req@ffff8b71f6796600 x1630377223958224/t146113932691(0) o36->e47c55d1-16e3-2d00-bb0a-bee0e04bc96f@10.8.1.17@o2ib6:20/0 lens 504/2888 e 0 to 0 dl 1554933140 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 14:52:36 2019][2778074.952150] LustreError: 118497:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5a8df27200 x1628639117476144/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:12/0 lens 448/440 e 0 to 0 dl 1554933162 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 14:53:07 2019][2778105.898594] LustreError: 118293:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554933097, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b747330cec0/0x53d0f214fc964c17 lrc: 3/1,0 mode: --/PR res: [0x2c0012124:0xe9bb:0x0].0x0 bits 0x13/0x8 rrc: 13 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118293 timeout: 0 lvb_type: 0 [Wed Apr 10 14:53:07 2019][2778105.938429] LustreError: 118293:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 14:53:20 2019][2778118.961861] LustreError: 118489:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554933110, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b782fe1cec0/0x53d0f214fcd635a9 lrc: 3/1,0 mode: --/PR res: [0x2c00128f9:0x180d:0x0].0x0 bits 0x13/0x8 rrc: 12 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118489 timeout: 0 lvb_type: 0 [Wed Apr 10 14:53:20 2019][2778119.001739] LustreError: 118489:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 14:54:57 2019][2778216.261875] LNet: Service thread pid 118293 was inactive for 200.36s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 14:54:57 2019][2778216.279078] Pid: 118293, comm: mdt02_034 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 14:54:57 2019][2778216.289094] Call Trace: [Wed Apr 10 14:54:57 2019][2778216.291734] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 14:54:57 2019][2778216.298877] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 14:54:57 2019][2778216.306264] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 14:54:57 2019][2778216.313278] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 14:54:57 2019][2778216.320489] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 14:54:57 2019][2778216.327160] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 14:54:57 2019][2778216.334338] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 14:54:57 2019][2778216.341100] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 14:54:57 2019][2778216.347754] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 14:54:57 2019][2778216.354706] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 14:54:57 2019][2778216.362021] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 14:54:57 2019][2778216.368392] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 14:54:57 2019][2778216.375508] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 14:54:57 2019][2778216.383436] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 14:54:57 2019][2778216.389981] [] kthread+0xd1/0xe0 [Wed Apr 10 14:54:57 2019][2778216.395076] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 14:54:57 2019][2778216.401725] [] 0xffffffffffffffff [Wed Apr 10 14:54:57 2019][2778216.406965] LustreError: dumping log to /tmp/lustre-log.1554933297.118293 [Wed Apr 10 14:54:58 2019][2778216.976221] Pid: 118391, comm: mdt01_099 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 14:54:58 2019][2778216.986220] Call Trace: [Wed Apr 10 14:54:58 2019][2778216.988862] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 14:54:58 2019][2778216.995997] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 14:54:58 2019][2778217.003367] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 14:54:58 2019][2778217.010308] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 14:54:58 2019][2778217.016983] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 14:54:58 2019][2778217.023206] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 14:54:58 2019][2778217.029949] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 14:54:58 2019][2778217.035840] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 14:54:58 2019][2778217.042970] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 14:54:58 2019][2778217.050912] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 14:54:58 2019][2778217.057428] [] kthread+0xd1/0xe0 [Wed Apr 10 14:54:58 2019][2778217.062520] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 14:54:58 2019][2778217.069181] [] 0xffffffffffffffff [Wed Apr 10 14:55:10 2019][2778229.574145] LNet: Service thread pid 118279 was inactive for 200.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 14:55:10 2019][2778229.591354] LNet: Skipped 1 previous similar message [Wed Apr 10 14:55:10 2019][2778229.596501] Pid: 118279, comm: mdt01_050 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 14:55:10 2019][2778229.606518] Call Trace: [Wed Apr 10 14:55:10 2019][2778229.609155] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 14:55:10 2019][2778229.616272] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 14:55:10 2019][2778229.623655] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 14:55:10 2019][2778229.630597] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 14:55:10 2019][2778229.637264] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 14:55:10 2019][2778229.643485] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 14:55:11 2019][2778229.650227] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 14:55:11 2019][2778229.656104] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 14:55:11 2019][2778229.663244] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 14:55:11 2019][2778229.671140] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 14:55:11 2019][2778229.677641] [] kthread+0xd1/0xe0 [Wed Apr 10 14:55:11 2019][2778229.682745] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 14:55:11 2019][2778229.689392] [] 0xffffffffffffffff [Wed Apr 10 14:55:11 2019][2778229.694588] LustreError: dumping log to /tmp/lustre-log.1554933311.118279 [Wed Apr 10 14:55:11 2019][2778229.742479] LNet: Service thread pid 118489 was inactive for 200.77s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 14:55:17 2019][2778235.669283] Lustre: fir-MDT0000: haven't heard from client 4d1e24f1-b9b4-9f49-84ea-8b9409e3d8eb (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4f8e54e000, cur 1554933317 expire 1554933167 last 1554933090 [Wed Apr 10 14:55:17 2019][2778235.691743] Lustre: Skipped 2 previous similar messages [Wed Apr 10 14:56:38 2019][2778317.021142] Lustre: fir-MDT0002: Connection restored to (at 10.9.103.11@o2ib4) [Wed Apr 10 14:56:38 2019][2778317.028649] Lustre: Skipped 741 previous similar messages [Wed Apr 10 14:56:50 2019][2778328.973486] Lustre: fir-MDT0002: Client 26c07b92-3e88-a582-abe6-28f16a4cf62b (at 10.9.108.26@o2ib4) reconnecting [Wed Apr 10 14:56:50 2019][2778328.983851] Lustre: Skipped 742 previous similar messages [Wed Apr 10 14:59:16 2019][2778474.674404] Lustre: fir-MDT0000: haven't heard from client 5a7ac3a2-9a33-8f4d-0bba-8552442e9a4c (at 10.9.101.23@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5902403800, cur 1554933556 expire 1554933406 last 1554933329 [Wed Apr 10 14:59:16 2019][2778474.696369] Lustre: Skipped 2 previous similar messages [Wed Apr 10 15:01:05 2019][2778583.889758] LustreError: 119044:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107a5:0x4a71:0x0] [Wed Apr 10 15:06:05 2019][2778883.682640] Lustre: fir-MDT0000: haven't heard from client af1f73a7-8062-08cb-5ec1-ad16b20895d4 (at 10.8.25.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b58d7a5ac00, cur 1554933965 expire 1554933815 last 1554933738 [Wed Apr 10 15:06:05 2019][2778883.704536] Lustre: Skipped 2 previous similar messages [Wed Apr 10 15:06:38 2019][2778917.494501] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 15:06:38 2019][2778917.502000] Lustre: Skipped 802 previous similar messages [Wed Apr 10 15:06:50 2019][2778929.085340] Lustre: fir-MDT0002: Client 40c21896-2b28-d1d6-5611-05f09dc83863 (at 10.9.108.24@o2ib4) reconnecting [Wed Apr 10 15:06:50 2019][2778929.095690] Lustre: Skipped 791 previous similar messages [Wed Apr 10 15:08:15 2019][2779014.182109] LustreError: 119003:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107f4:0xa503:0x0] [Wed Apr 10 15:16:39 2019][2779517.703845] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Wed Apr 10 15:16:39 2019][2779517.711160] Lustre: Skipped 795 previous similar messages [Wed Apr 10 15:16:50 2019][2779529.419509] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Wed Apr 10 15:16:50 2019][2779529.429690] Lustre: Skipped 790 previous similar messages [Wed Apr 10 15:18:50 2019][2779649.019013] Lustre: 118361:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Apr 10 15:19:32 2019][2779690.699254] Lustre: fir-MDT0002: haven't heard from client a113795d-26c3-ae3f-6fff-6a09176dbfb5 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5888157c00, cur 1554934772 expire 1554934622 last 1554934545 [Wed Apr 10 15:19:32 2019][2779690.721138] Lustre: Skipped 8 previous similar messages [Wed Apr 10 15:19:35 2019][2779694.122070] Lustre: 118380:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Apr 10 15:19:47 2019][2779706.553395] Lustre: 118380:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Apr 10 15:26:39 2019][2780118.053201] Lustre: fir-MDT0002: Connection restored to (at 10.9.107.48@o2ib4) [Wed Apr 10 15:26:39 2019][2780118.060699] Lustre: Skipped 796 previous similar messages [Wed Apr 10 15:26:52 2019][2780130.815227] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Wed Apr 10 15:26:52 2019][2780130.825320] Lustre: Skipped 796 previous similar messages [Wed Apr 10 15:31:36 2019][2780415.503023] Lustre: 118256:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554935489/real 1554935489] req@ffff8b61e0ea2700 x1630438838900864/t0(0) o104->fir-MDT0002@10.8.26.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554935496 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 15:31:36 2019][2780415.530636] Lustre: 118256:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [Wed Apr 10 15:31:44 2019][2780423.551187] Lustre: 118351:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 15:31:44 2019][2780423.551187] req@ffff8b6c94b73c00 x1629828709265728/t0(0) o101->a0371f67-4be2-4192-d3f8-7917466cd048@10.8.21.7@o2ib6:19/0 lens 1784/3288 e 1 to 0 dl 1554935509 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 15:31:44 2019][2780423.580354] Lustre: 118351:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 15:31:45 2019][2780424.081199] Lustre: 118227:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 15:31:45 2019][2780424.081199] req@ffff8b51b6e85100 x1628558434644016/t0(0) o101->e458857b-15c5-7752-4466-6b97eda69ac1@10.8.12.24@o2ib6:20/0 lens 576/3264 e 1 to 0 dl 1554935510 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 15:31:45 2019][2780424.110380] Lustre: 118227:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 15:31:46 2019][2780425.504227] Lustre: 118245:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 15:31:46 2019][2780425.504227] req@ffff8b7646665a00 x1628591775918048/t0(0) o101->91177ecb-3af9-b598-0df4-bd163eca9e44@10.8.26.3@o2ib6:21/0 lens 576/3264 e 1 to 0 dl 1554935511 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 15:31:46 2019][2780425.533300] Lustre: 118245:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages [Wed Apr 10 15:31:56 2019][2780435.121428] Lustre: 118361:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 15:31:56 2019][2780435.121428] req@ffff8b530e80d100 x1628581332906560/t0(0) o101->41459fb7-0754-40e6-eaf6-d1508509a127@10.8.25.23@o2ib6:1/0 lens 576/3264 e 0 to 0 dl 1554935521 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 15:31:56 2019][2780435.150608] Lustre: 118361:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages [Wed Apr 10 15:32:11 2019][2780450.541750] Lustre: 118256:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554935524/real 1554935524] req@ffff8b61e0ea2700 x1630438838900864/t0(0) o104->fir-MDT0002@10.8.26.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554935531 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 15:32:11 2019][2780450.569181] Lustre: 118256:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Wed Apr 10 15:32:25 2019][2780463.715148] Lustre: fir-MDT0000: haven't heard from client a123b1b5-b6d5-77a3-e03c-ed2ee4deefa6 (at 10.8.26.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b504c1ce400, cur 1554935545 expire 1554935395 last 1554935318 [Wed Apr 10 15:32:25 2019][2780463.736950] Lustre: Skipped 14 previous similar messages [Wed Apr 10 15:36:40 2019][2780718.852416] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Wed Apr 10 15:36:40 2019][2780718.862769] Lustre: Skipped 853 previous similar messages [Wed Apr 10 15:36:53 2019][2780732.427763] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 15:36:53 2019][2780732.438115] Lustre: Skipped 846 previous similar messages [Wed Apr 10 15:37:39 2019][2780778.173102] LustreError: 118672:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001069d:0x98e:0x0] [Wed Apr 10 15:46:41 2019][2781320.531843] Lustre: fir-MDT0002: Connection restored to (at 10.9.103.11@o2ib4) [Wed Apr 10 15:46:41 2019][2781320.539338] Lustre: Skipped 793 previous similar messages [Wed Apr 10 15:46:55 2019][2781334.030970] Lustre: fir-MDT0002: Client e18a93c6-a11b-0b52-af65-1483de6beb82 (at 10.8.8.18@o2ib6) reconnecting [Wed Apr 10 15:46:55 2019][2781334.041154] Lustre: Skipped 795 previous similar messages [Wed Apr 10 15:48:17 2019][2781416.033699] Lustre: 118172:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0000: Failure to clear the changelog for user 1: -22 [Wed Apr 10 15:49:21 2019][2781479.852951] Lustre: 118568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554936554/real 1554936554] req@ffff8b713423ad00 x1630438863794352/t0(0) o104->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1554936561 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 15:49:21 2019][2781479.880476] Lustre: 118568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [Wed Apr 10 15:49:29 2019][2781488.639119] Lustre: 118245:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 15:49:29 2019][2781488.639119] req@ffff8b71b76cd700 x1628763864019456/t0(0) o36->b14dc57b-b6ae-736a-c8c8-b28de0f48811@10.9.104.57@o2ib4:4/0 lens 504/2888 e 1 to 0 dl 1554936574 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 15:49:29 2019][2781488.668221] Lustre: 118245:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 15:49:35 2019][2781493.890238] Lustre: 118568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554936568/real 1554936568] req@ffff8b713423ad00 x1630438863794352/t0(0) o104->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1554936575 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 15:49:35 2019][2781493.917784] Lustre: 118568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Wed Apr 10 15:49:49 2019][2781507.927548] LustreError: 118568:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.114.5@o2ib4) failed to reply to blocking AST (req@ffff8b713423ad00 x1630438863794352 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b6ce32606c0/0x53d0f2152d522489 lrc: 4/0,0 mode: PR/PR res: [0x2c00107f4:0xac68:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.9.114.5@o2ib4 remote: 0xc9c50a471a31258c expref: 2764 pid: 118370 timeout: 2781467 lvb_type: 0 [Wed Apr 10 15:49:49 2019][2781507.970876] LustreError: 118568:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message [Wed Apr 10 15:49:49 2019][2781507.981153] LustreError: 138-a: fir-MDT0002: A client on nid 10.9.114.5@o2ib4 was evicted due to a lock blocking callback time out: rc -110 [Wed Apr 10 15:49:49 2019][2781507.993855] LustreError: Skipped 1 previous similar message [Wed Apr 10 15:49:49 2019][2781507.999654] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.9.114.5@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8b6ce32606c0/0x53d0f2152d522489 lrc: 3/0,0 mode: PR/PR res: [0x2c00107f4:0xac68:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.9.114.5@o2ib4 remote: 0xc9c50a471a31258c expref: 2765 pid: 118370 timeout: 0 lvb_type: 0 [Wed Apr 10 15:49:49 2019][2781508.037403] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message [Wed Apr 10 15:52:47 2019][2781685.740231] Lustre: fir-MDT0000: haven't heard from client 73e4334b-e1a4-a4d7-284d-1cd9fb4562bc (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4f4d4a5000, cur 1554936767 expire 1554936617 last 1554936540 [Wed Apr 10 15:52:47 2019][2781685.762134] Lustre: Skipped 8 previous similar messages [Wed Apr 10 15:56:44 2019][2781922.923180] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Wed Apr 10 15:56:44 2019][2781922.930603] Lustre: Skipped 805 previous similar messages [Wed Apr 10 15:56:57 2019][2781936.060656] Lustre: fir-MDT0002: Client e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6) reconnecting [Wed Apr 10 15:56:57 2019][2781936.070835] Lustre: Skipped 798 previous similar messages [Wed Apr 10 16:03:24 2019][2782322.753357] Lustre: fir-MDT0000: haven't heard from client 4396a612-edbd-dc95-0827-14aad3fc5f61 (at 10.9.104.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b595ef12000, cur 1554937404 expire 1554937254 last 1554937177 [Wed Apr 10 16:03:24 2019][2782322.775317] Lustre: Skipped 1 previous similar message [Wed Apr 10 16:06:46 2019][2782525.157781] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Wed Apr 10 16:06:46 2019][2782525.165093] Lustre: Skipped 812 previous similar messages [Wed Apr 10 16:07:00 2019][2782538.746982] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 16:07:00 2019][2782538.757338] Lustre: Skipped 791 previous similar messages [Wed Apr 10 16:12:41 2019][2782879.764818] Lustre: fir-MDT0000: haven't heard from client 87933400-52b6-d891-f931-efda0e6b6639 (at 10.9.104.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8cc919f400, cur 1554937961 expire 1554937811 last 1554937734 [Wed Apr 10 16:12:41 2019][2782879.786783] Lustre: Skipped 2 previous similar messages [Wed Apr 10 16:12:58 2019][2782897.743568] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Wed Apr 10 16:15:54 2019][2783073.642919] LustreError: 119091:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b6dd8a6b900 x1628639128664704/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:0/0 lens 448/440 e 0 to 0 dl 1554938160 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 16:16:48 2019][2783126.867996] Lustre: fir-MDT0002: Connection restored to (at 10.9.103.11@o2ib4) [Wed Apr 10 16:16:48 2019][2783126.875493] Lustre: Skipped 791 previous similar messages [Wed Apr 10 16:17:02 2019][2783141.168814] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Wed Apr 10 16:17:02 2019][2783141.179081] Lustre: Skipped 799 previous similar messages [Wed Apr 10 16:25:46 2019][2783664.788368] Lustre: MGS: haven't heard from client b452493f-dcad-f47f-559b-e54fd147afb0 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8656240c00, cur 1554938746 expire 1554938596 last 1554938519 [Wed Apr 10 16:25:46 2019][2783664.809475] Lustre: Skipped 2 previous similar messages [Wed Apr 10 16:26:50 2019][2783729.278462] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Wed Apr 10 16:26:50 2019][2783729.285872] Lustre: Skipped 800 previous similar messages [Wed Apr 10 16:27:04 2019][2783743.318476] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Wed Apr 10 16:27:04 2019][2783743.328567] Lustre: Skipped 796 previous similar messages [Wed Apr 10 16:28:04 2019][2783803.040085] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 16:28:04 2019][2783803.057450] LustreError: Skipped 935 previous similar messages [Wed Apr 10 16:28:36 2019][2783835.615507] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 16:32:21 2019][2784060.045173] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554939134/real 1554939134] req@ffff8b50acb27800 x1630438894556528/t0(0) o106->fir-MDT0002@10.8.24.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554939141 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 16:32:21 2019][2784060.072700] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Wed Apr 10 16:32:28 2019][2784067.082311] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554939141/real 1554939141] req@ffff8b50acb27800 x1630438894556528/t0(0) o106->fir-MDT0002@10.8.24.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554939148 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 16:32:29 2019][2784068.086340] Lustre: 118421:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 16:32:29 2019][2784068.086340] req@ffff8b5a8e7ed700 x1629246705203280/t0(0) o101->bafe047a-89ea-21d5-a24e-5c407648cfc7@10.0.10.3@o2ib7:4/0 lens 480/568 e 1 to 0 dl 1554939154 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 16:32:35 2019][2784074.109458] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554939148/real 1554939148] req@ffff8b50acb27800 x1630438894556528/t0(0) o106->fir-MDT0002@10.8.24.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554939155 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 16:32:49 2019][2784088.137749] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554939162/real 1554939162] req@ffff8b50acb27800 x1630438894556528/t0(0) o106->fir-MDT0002@10.8.24.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554939169 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 16:32:49 2019][2784088.165287] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Wed Apr 10 16:33:03 2019][2784102.642925] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 16:33:10 2019][2784109.176186] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554939183/real 1554939183] req@ffff8b50acb27800 x1630438894556528/t0(0) o106->fir-MDT0002@10.8.24.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554939190 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 16:33:10 2019][2784109.203702] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Wed Apr 10 16:33:23 2019][2784121.881684] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 16:33:23 2019][2784121.899054] LustreError: Skipped 1 previous similar message [Wed Apr 10 16:33:31 2019][2784130.277248] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 16:33:45 2019][2784144.214913] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554939218/real 1554939218] req@ffff8b50acb27800 x1630438894556528/t0(0) o106->fir-MDT0002@10.8.24.29@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554939225 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 16:33:45 2019][2784144.242446] Lustre: 118224:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Wed Apr 10 16:34:35 2019][2784194.714075] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 16:36:52 2019][2784331.440508] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Wed Apr 10 16:36:52 2019][2784331.450862] Lustre: Skipped 817 previous similar messages [Wed Apr 10 16:37:04 2019][2784343.725224] Lustre: fir-MDT0002: Client 64821009-3ac8-89b2-b714-0e5d4d7937f9 (at 10.9.101.72@o2ib4) reconnecting [Wed Apr 10 16:37:04 2019][2784343.735578] Lustre: Skipped 807 previous similar messages [Wed Apr 10 16:39:02 2019][2784461.024863] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 16:39:42 2019][2784500.798647] Lustre: fir-MDT0000: haven't heard from client 39b21bb1-06bc-8c69-c9c3-5986c42b070c (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8c3440b000, cur 1554939582 expire 1554939432 last 1554939355 [Wed Apr 10 16:39:42 2019][2784500.820455] Lustre: Skipped 9 previous similar messages [Wed Apr 10 16:40:47 2019][2784566.738805] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 16:40:47 2019][2784566.756179] LustreError: Skipped 1 previous similar message [Wed Apr 10 16:40:55 2019][2784574.509146] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 16:42:35 2019][2784673.848680] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 16:42:48 2019][2784687.279885] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Wed Apr 10 16:43:00 2019][2784699.121091] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 16:43:00 2019][2784699.138485] LustreError: Skipped 1 previous similar message [Wed Apr 10 16:45:40 2019][2784859.514396] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 16:45:40 2019][2784859.524665] Lustre: Skipped 1 previous similar message [Wed Apr 10 16:46:37 2019][2784916.115382] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 16:46:53 2019][2784932.373530] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Wed Apr 10 16:46:53 2019][2784932.381034] Lustre: Skipped 809 previous similar messages [Wed Apr 10 16:46:57 2019][2784936.080293] Lustre: 118424:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 16:46:57 2019][2784936.080293] req@ffff8b7693f18c00 x1629292586433296/t0(0) o101->99b2b05a-80e4-edd2-5271-b136b8e4528d@10.9.108.67@o2ib4:2/0 lens 616/3264 e 1 to 0 dl 1554940022 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 16:46:58 2019][2784937.082318] Lustre: 118716:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 16:46:58 2019][2784937.082318] req@ffff8b8cb1fb2700 x1628597183478224/t0(0) o101->99abaf9a-6d16-bfe7-2d5c-9e199780b8b7@10.9.101.71@o2ib4:3/0 lens 616/3264 e 1 to 0 dl 1554940023 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 16:46:58 2019][2784937.111481] Lustre: 118716:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Wed Apr 10 16:46:59 2019][2784938.292340] Lustre: 117464:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/3), not sending early reply [Wed Apr 10 16:46:59 2019][2784938.292340] req@ffff8b61ffb5f500 x1628870333820208/t0(0) o101->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:4/0 lens 576/3264 e 1 to 0 dl 1554940024 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 16:46:59 2019][2784938.321509] Lustre: 117464:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 16:47:05 2019][2784944.088729] Lustre: fir-MDT0002: Client bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83 (at 10.8.26.21@o2ib6) reconnecting [Wed Apr 10 16:47:05 2019][2784944.098992] Lustre: Skipped 811 previous similar messages [Wed Apr 10 16:47:07 2019][2784946.220518] Lustre: 118682:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 16:47:07 2019][2784946.220518] req@ffff8b868477ec00 x1628638571857376/t0(0) o101->67753486-aa02-1106-6851-77990d7e25c7@10.9.107.61@o2ib4:12/0 lens 616/3264 e 0 to 0 dl 1554940032 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 16:47:16 2019][2784955.779700] Lustre: 118345:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554940029/real 1554940029] req@ffff8b6483313f00 x1630438908309920/t0(0) o104->fir-MDT0002@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554940036 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 16:47:17 2019][2784955.807129] Lustre: 118345:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Wed Apr 10 16:48:12 2019][2785011.699850] LustreError: 118467:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554940002, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b5442788240/0x53d0f215687da188 lrc: 3/1,0 mode: --/PR res: [0x2c001083f:0xb66e:0x0].0x0 bits 0x13/0x8 rrc: 16 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118467 timeout: 0 lvb_type: 0 [Wed Apr 10 16:48:12 2019][2785011.739685] LustreError: 118467:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Wed Apr 10 16:48:59 2019][2785058.532478] LustreError: 118597:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010750:0x5168:0x0] [Wed Apr 10 16:50:03 2019][2785122.260133] LNet: Service thread pid 118482 was inactive for 200.49s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 16:50:03 2019][2785122.277334] Pid: 118482, comm: mdt00_091 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 16:50:03 2019][2785122.287331] Call Trace: [Wed Apr 10 16:50:03 2019][2785122.289973] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 16:50:03 2019][2785122.297102] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 16:50:03 2019][2785122.304477] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 16:50:03 2019][2785122.311480] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 16:50:03 2019][2785122.318784] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Wed Apr 10 16:50:03 2019][2785122.325966] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 16:50:03 2019][2785122.332921] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 16:50:03 2019][2785122.339847] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 16:50:03 2019][2785122.346790] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 16:50:03 2019][2785122.354099] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 16:50:03 2019][2785122.360447] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 16:50:03 2019][2785122.367563] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 16:50:03 2019][2785122.375458] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 16:50:03 2019][2785122.381968] [] kthread+0xd1/0xe0 [Wed Apr 10 16:50:03 2019][2785122.387055] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 16:50:03 2019][2785122.393701] [] 0xffffffffffffffff [Wed Apr 10 16:50:03 2019][2785122.398895] LustreError: dumping log to /tmp/lustre-log.1554940203.118482 [Wed Apr 10 16:50:04 2019][2785123.781281] LNet: Service thread pid 118543 was inactive for 202.01s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 16:50:04 2019][2785123.798482] Pid: 118543, comm: mdt03_052 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 16:50:05 2019][2785123.808488] Call Trace: [Wed Apr 10 16:50:05 2019][2785123.811128] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.818248] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.825623] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 16:50:05 2019][2785123.832635] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 16:50:05 2019][2785123.839813] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Wed Apr 10 16:50:05 2019][2785123.846986] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 16:50:05 2019][2785123.853730] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 16:50:05 2019][2785123.860386] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.867329] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.874631] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.880998] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.888130] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.896024] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.902541] [] kthread+0xd1/0xe0 [Wed Apr 10 16:50:05 2019][2785123.907631] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 16:50:05 2019][2785123.914278] [] 0xffffffffffffffff [Wed Apr 10 16:50:05 2019][2785123.919474] Pid: 118156, comm: mdt01_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 16:50:05 2019][2785123.929473] Call Trace: [Wed Apr 10 16:50:05 2019][2785123.932121] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.939577] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.946960] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 16:50:05 2019][2785123.953963] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 16:50:05 2019][2785123.961138] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Wed Apr 10 16:50:05 2019][2785123.968316] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 16:50:05 2019][2785123.975057] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 16:50:05 2019][2785123.981715] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.988646] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785123.995939] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.002290] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.009413] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.017310] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.023817] [] kthread+0xd1/0xe0 [Wed Apr 10 16:50:05 2019][2785124.028907] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 16:50:05 2019][2785124.035554] [] 0xffffffffffffffff [Wed Apr 10 16:50:05 2019][2785124.040753] Pid: 118667, comm: mdt03_078 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 16:50:05 2019][2785124.050749] Call Trace: [Wed Apr 10 16:50:05 2019][2785124.053390] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.060514] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.067891] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 16:50:05 2019][2785124.074932] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 16:50:05 2019][2785124.082111] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Wed Apr 10 16:50:05 2019][2785124.089286] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 16:50:05 2019][2785124.096029] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 16:50:05 2019][2785124.102687] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.109621] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.116911] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.123243] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.130368] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.138257] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.144766] [] kthread+0xd1/0xe0 [Wed Apr 10 16:50:05 2019][2785124.149872] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 16:50:05 2019][2785124.156525] [] 0xffffffffffffffff [Wed Apr 10 16:50:05 2019][2785124.161735] Pid: 118168, comm: mdt00_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 16:50:05 2019][2785124.171731] Call Trace: [Wed Apr 10 16:50:05 2019][2785124.174383] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.181511] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.188879] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 16:50:05 2019][2785124.195805] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 16:50:05 2019][2785124.202459] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 16:50:05 2019][2785124.208681] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 16:50:05 2019][2785124.215415] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 16:50:05 2019][2785124.221292] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.228415] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.236303] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 16:50:05 2019][2785124.242802] [] kthread+0xd1/0xe0 [Wed Apr 10 16:50:05 2019][2785124.247891] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 16:50:05 2019][2785124.254539] [] 0xffffffffffffffff [Wed Apr 10 16:50:05 2019][2785124.259735] LNet: Service thread pid 118287 was inactive for 202.53s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 16:56:53 2019][2785532.389003] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.18@o2ib4) [Wed Apr 10 16:56:53 2019][2785532.396509] Lustre: Skipped 1099 previous similar messages [Wed Apr 10 16:57:05 2019][2785544.390882] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Wed Apr 10 16:57:05 2019][2785544.401227] Lustre: Skipped 1085 previous similar messages [Wed Apr 10 16:57:38 2019][2785576.820420] Lustre: fir-MDT0000: haven't heard from client 12994599-ad9c-a9cf-bbe3-80e2e4fd455c (at 10.9.106.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b69d7bbe800, cur 1554940658 expire 1554940508 last 1554940431 [Wed Apr 10 16:57:38 2019][2785576.842425] Lustre: Skipped 3 previous similar messages [Wed Apr 10 16:59:46 2019][2785705.689724] LustreError: 118783:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010959:0xecb:0x0] [Wed Apr 10 17:02:44 2019][2785882.952672] Lustre: 118588:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554940957/real 1554940957] req@ffff8b722d3a6000 x1630438931846176/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554940964 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 17:02:51 2019][2785889.979824] Lustre: 118588:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554940964/real 1554940964] req@ffff8b722d3a6000 x1630438931846176/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554940971 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 17:02:52 2019][2785891.539857] Lustre: 118488:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 17:02:52 2019][2785891.539857] req@ffff8b6f2d739e00 x1630348648500176/t0(0) o101->04a140c3-a482-7014-c768-6a1fe4cf0c6b@10.9.112.13@o2ib4:27/0 lens 376/1600 e 1 to 0 dl 1554940977 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:02:52 2019][2785891.569119] Lustre: 118488:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Wed Apr 10 17:02:58 2019][2785897.006956] Lustre: 118588:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554940971/real 1554940971] req@ffff8b722d3a6000 x1630438931846176/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554940978 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 17:03:05 2019][2785904.035100] Lustre: 118588:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554940978/real 1554940978] req@ffff8b722d3a6000 x1630438931846176/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554940985 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 17:03:19 2019][2785918.063382] Lustre: 118588:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554940992/real 1554940992] req@ffff8b722d3a6000 x1630438931846176/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554940999 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 17:03:19 2019][2785918.090741] Lustre: 118588:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Wed Apr 10 17:03:40 2019][2785939.088809] Lustre: 117906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554941013/real 1554941013] req@ffff8b6c180af500 x1630438933001728/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554941020 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 17:03:40 2019][2785939.116165] Lustre: 117906:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Wed Apr 10 17:03:51 2019][2785950.151038] Lustre: 118207:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 17:03:51 2019][2785950.151038] req@ffff8b6c24da4e00 x1630434792574448/t0(0) o101->1fed4b9d-ca5d-165d-887c-2f1234134d0c@10.8.9.10@o2ib6:26/0 lens 480/568 e 0 to 0 dl 1554941036 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:04:15 2019][2785974.101536] Lustre: 118588:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554941048/real 1554941048] req@ffff8b722d3a6000 x1630438931846176/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554941055 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 17:04:15 2019][2785974.128883] Lustre: 118588:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [Wed Apr 10 17:04:34 2019][2785993.521926] Lustre: 118281:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 17:04:34 2019][2785993.521926] req@ffff8b5e4d3d8300 x1629293837077984/t0(0) o36->cbd6d5ca-c880-41ab-b215-45c09cf901af@10.8.8.37@o2ib6:9/0 lens 520/2888 e 0 to 0 dl 1554941079 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:05:11 2019][2786030.139708] LustreError: 118588:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.8@o2ib6) failed to reply to blocking AST (req@ffff8b722d3a6000 x1630438931846176 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8b63d1ba9680/0x53d0f2157b03e464 lrc: 4/0,0 mode: PR/PR res: [0x200011d0f:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.9.8@o2ib6 remote: 0x8c0f350dbe038616 expref: 535095 pid: 118152 timeout: 2786109 lvb_type: 0 [Wed Apr 10 17:05:11 2019][2786030.182696] LustreError: 138-a: fir-MDT0000: A client on nid 10.8.9.8@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [Wed Apr 10 17:05:11 2019][2786030.195257] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.9.8@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8b63d1ba9680/0x53d0f2157b03e464 lrc: 3/0,0 mode: PR/PR res: [0x200011d0f:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.9.8@o2ib6 remote: 0x8c0f350dbe038616 expref: 535096 pid: 118152 timeout: 0 lvb_type: 0 [Wed Apr 10 17:05:11 2019][2786030.233750] Lustre: 118406:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (61:1s); client may timeout. req@ffff8b5e4d3d8300 x1629293837077984/t154985701228(0) o36->cbd6d5ca-c880-41ab-b215-45c09cf901af@10.8.8.37@o2ib6:9/0 lens 520/424 e 0 to 0 dl 1554941110 ref 1 fl Complete:/0/0 rc 0/0 [Wed Apr 10 17:05:57 2019][2786076.647628] LNet: Service thread pid 118588 was inactive for 200.69s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 17:05:57 2019][2786076.664823] LNet: Skipped 3 previous similar messages [Wed Apr 10 17:05:57 2019][2786076.670061] Pid: 118588, comm: mdt02_109 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 17:05:57 2019][2786076.680068] Call Trace: [Wed Apr 10 17:05:57 2019][2786076.682713] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Wed Apr 10 17:05:57 2019][2786076.689841] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 17:05:57 2019][2786076.697226] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 17:05:57 2019][2786076.704245] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 17:05:57 2019][2786076.711431] [] mdt_layout_change+0x2a4/0x430 [mdt] [Wed Apr 10 17:05:57 2019][2786076.718095] [] mdt_intent_layout+0x7ee/0xcc0 [mdt] [Wed Apr 10 17:05:57 2019][2786076.724762] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 17:05:57 2019][2786076.731433] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 17:05:57 2019][2786076.738384] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 17:05:57 2019][2786076.745707] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 17:05:57 2019][2786076.752061] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 17:05:57 2019][2786076.759211] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 17:05:57 2019][2786076.767123] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 17:05:57 2019][2786076.773652] [] kthread+0xd1/0xe0 [Wed Apr 10 17:05:57 2019][2786076.778765] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 17:05:58 2019][2786076.785427] [] 0xffffffffffffffff [Wed Apr 10 17:05:58 2019][2786076.790640] LustreError: dumping log to /tmp/lustre-log.1554941157.118588 [Wed Apr 10 17:06:41 2019][2786120.196523] LustreError: 118588:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554941111, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b725befaac0/0x53d0f2158721a921 lrc: 3/0,1 mode: --/EX res: [0x200011d0f:0x4a:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118588 timeout: 0 lvb_type: 0 [Wed Apr 10 17:06:41 2019][2786120.236014] LustreError: 118588:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 9 previous similar messages [Wed Apr 10 17:06:53 2019][2786132.470941] Lustre: fir-MDT0002: Connection restored to bcead966-3991-3be5-a9ca-eba502e01069 (at 10.9.108.31@o2ib4) [Wed Apr 10 17:06:53 2019][2786132.481588] Lustre: Skipped 1111 previous similar messages [Wed Apr 10 17:07:05 2019][2786144.734331] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Wed Apr 10 17:07:05 2019][2786144.744591] Lustre: Skipped 1111 previous similar messages [Wed Apr 10 17:07:20 2019][2786159.427119] LustreError: 117467:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b5e8f2ee000 x1630438936871328/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Apr 10 17:07:45 2019][2786184.523847] Lustre: 118254:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 17:07:45 2019][2786184.523847] req@ffff8b5acfb9dd00 x1630383748384048/t0(0) o36->10a04728-b83a-e0e8-706e-f82508c6877b@10.8.29.3@o2ib6:20/0 lens 536/2888 e 0 to 0 dl 1554941270 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:07:45 2019][2786184.552946] Lustre: 118254:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 17:07:53 2019][2786192.171534] LNet: Service thread pid 118588 completed after 316.21s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Apr 10 17:08:04 2019][2786203.001789] LustreError: 118430:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b5848f03000 x1630438937680960/t0(0) o104->fir-MDT0000@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Apr 10 17:08:29 2019][2786228.122746] Lustre: 118536:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 17:08:29 2019][2786228.122746] req@ffff8b5481aac500 x1630383748389792/t0(0) o36->10a04728-b83a-e0e8-706e-f82508c6877b@10.8.29.3@o2ib6:4/0 lens 536/2888 e 0 to 0 dl 1554941314 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:09:49 2019][2786308.081687] LustreError: 117480:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b71eef68000 x1628639135536784/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:25/0 lens 448/440 e 0 to 0 dl 1554941395 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:11:00 2019][2786378.836966] Lustre: fir-MDT0002: haven't heard from client c65ff351-a3e9-9dcc-ef16-10d61a063f1a (at 10.8.16.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7cfefed800, cur 1554941460 expire 1554941310 last 1554941233 [Wed Apr 10 17:11:00 2019][2786378.858760] Lustre: Skipped 7 previous similar messages [Wed Apr 10 17:12:22 2019][2786460.857651] Lustre: 118282:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554941535/real 1554941535] req@ffff8b4ff37b0f00 x1630438943134160/t0(0) o106->fir-MDT0002@10.8.22.28@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554941542 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 17:12:22 2019][2786460.885168] Lustre: 118282:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [Wed Apr 10 17:12:40 2019][2786479.002031] Lustre: 118260:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 17:12:40 2019][2786479.002031] req@ffff8b4eb89ec800 x1629928217642560/t0(0) o101->d688f6d7-017e-8817-dbcd-6b016c3633d8@10.9.0.61@o2ib4:15/0 lens 480/568 e 0 to 0 dl 1554941565 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:12:40 2019][2786479.031137] Lustre: 118260:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 17:13:05 2019][2786504.239190] Lustre: 118537:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Apr 10 17:16:54 2019][2786732.865922] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Wed Apr 10 17:16:54 2019][2786732.873328] Lustre: Skipped 1101 previous similar messages [Wed Apr 10 17:17:08 2019][2786746.880506] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Wed Apr 10 17:17:08 2019][2786746.890596] Lustre: Skipped 1093 previous similar messages [Wed Apr 10 17:23:43 2019][2787141.852907] Lustre: fir-MDT0000: haven't heard from client 6ffc2e8b-b062-5b88-97a9-a61177559c11 (at 10.9.101.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b88fca03400, cur 1554942223 expire 1554942073 last 1554941996 [Wed Apr 10 17:23:43 2019][2787141.874804] Lustre: Skipped 65 previous similar messages [Wed Apr 10 17:26:56 2019][2787335.006870] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Wed Apr 10 17:26:56 2019][2787335.017221] Lustre: Skipped 1092 previous similar messages [Wed Apr 10 17:27:09 2019][2787348.625336] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 17:27:09 2019][2787348.635696] Lustre: Skipped 1094 previous similar messages [Wed Apr 10 17:31:31 2019][2787610.114090] LustreError: 119113:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6ac7f20c00 x1628758068679920/t0(0) o37->91934493-be40-5b23-a558-2943654113ab@10.8.27.16@o2ib6:7/0 lens 448/440 e 0 to 0 dl 1554942697 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:33:21 2019][2787720.179248] LustreError: 118812:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011150:0x11bff:0x0] [Wed Apr 10 17:34:39 2019][2787797.866675] Lustre: fir-MDT0000: haven't heard from client 09ccae74-da49-3ed9-15f5-3662d8bb22c0 (at 10.9.103.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5902403400, cur 1554942879 expire 1554942729 last 1554942652 [Wed Apr 10 17:34:39 2019][2787797.888570] Lustre: Skipped 2 previous similar messages [Wed Apr 10 17:36:56 2019][2787935.239842] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.60@o2ib4) [Wed Apr 10 17:36:56 2019][2787935.247338] Lustre: Skipped 1135 previous similar messages [Wed Apr 10 17:37:11 2019][2787950.829078] Lustre: fir-MDT0002: Client e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6) reconnecting [Wed Apr 10 17:37:12 2019][2787950.839273] Lustre: Skipped 1097 previous similar messages [Wed Apr 10 17:38:52 2019][2788051.703402] LustreError: 118972:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010497:0x1330:0x0] [Wed Apr 10 17:38:52 2019][2788051.799262] LustreError: 118916:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011207:0x38f1:0x0] [Wed Apr 10 17:40:09 2019][2788128.508203] LustreError: 119218:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128f3:0xa934:0x0] [Wed Apr 10 17:40:46 2019][2788164.927327] Lustre: 118210:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554943239/real 1554943239] req@ffff8b5aec2a8600 x1630438983520096/t0(0) o106->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554943246 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 17:40:46 2019][2788164.954877] Lustre: 118210:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [Wed Apr 10 17:40:48 2019][2788166.876085] LustreError: 117559:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b8b9f83ce00 x1630438983706960/t0(0) o104->fir-MDT0002@10.9.114.5@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Apr 10 17:46:17 2019][2788496.482253] Lustre: 118153:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 17:46:17 2019][2788496.482253] req@ffff8b5499df4500 x1628659383656432/t0(0) o36->8f523a71-dbaa-8a10-c21b-13f39a091fa7@10.9.102.10@o2ib4:22/0 lens 600/2888 e 1 to 0 dl 1554943582 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:46:17 2019][2788496.511431] Lustre: 118153:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 17:46:21 2019][2788499.978307] Lustre: 118312:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 17:46:21 2019][2788499.978307] req@ffff8b71e1a59800 x1630473680274368/t0(0) o36->539bc8c3-974e-613c-c51e-9ed2d7677a6d@10.9.101.2@o2ib4:26/0 lens 600/2888 e 1 to 0 dl 1554943586 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:46:37 2019][2788516.546665] Lustre: 118210:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 17:46:37 2019][2788516.546665] req@ffff8b55be987500 x1630380198699184/t0(0) o36->26eab1fe-82e3-157a-2922-6187f62cdd31@10.9.108.47@o2ib4:12/0 lens 600/2888 e 1 to 0 dl 1554943602 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:46:56 2019][2788535.285140] Lustre: fir-MDT0002: Connection restored to 3dde01db-8bab-e6a5-6af0-a17c56d94df2 (at 10.9.101.66@o2ib4) [Wed Apr 10 17:46:56 2019][2788535.295747] Lustre: Skipped 1116 previous similar messages [Wed Apr 10 17:47:14 2019][2788553.239346] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Wed Apr 10 17:47:14 2019][2788553.249436] Lustre: Skipped 1100 previous similar messages [Wed Apr 10 17:47:22 2019][2788561.719606] Lustre: 118352:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 17:47:22 2019][2788561.719606] req@ffff8b72fc2b2700 x1629179824533216/t0(0) o36->39b21bb1-06bc-8c69-c9c3-5986c42b070c@10.8.0.65@o2ib6:27/0 lens 528/2888 e 0 to 0 dl 1554943647 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:47:32 2019][2788571.413811] LustreError: 118260:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943562, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b5143aeda00/0x53d0f215b3ffdec9 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 19 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118260 timeout: 0 lvb_type: 0 [Wed Apr 10 17:47:32 2019][2788571.439850] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Apr 10 17:47:32 2019][2788571.439870] LustreError: 118297:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943562, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b6c6450ec00/0x53d0f215b3fff367 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f215b3fff391 expref: -99 pid: 118297 timeout: 0 lvb_type: 0 [Wed Apr 10 17:47:32 2019][2788571.439995] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Apr 10 17:47:32 2019][2788571.439997] Lustre: Skipped 1 previous similar message [Wed Apr 10 17:47:32 2019][2788571.834216] LustreError: 119194:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00111d1:0x2a31:0x0] [Wed Apr 10 17:47:36 2019][2788575.479899] LustreError: 118576:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943566, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b771b345a00/0x53d0f215b410866a lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 19 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118576 timeout: 0 lvb_type: 0 [Wed Apr 10 17:47:39 2019][2788578.333957] LustreError: 118315:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943569, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b59a0663a80/0x53d0f215b41db23a lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f215b41db24f expref: -99 pid: 118315 timeout: 0 lvb_type: 0 [Wed Apr 10 17:47:41 2019][2788580.830277] Lustre: fir-MDT0000: Received new LWP connection from 10.0.10.52@o2ib7, removing former export from same NID [Wed Apr 10 17:47:42 2019][2788580.883013] Lustre: fir-MDT0002: haven't heard from client 4761d91d-2dfe-5702-4fe8-b846b6ec7501 (at 10.9.103.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b785fe9a000, cur 1554943662 expire 1554943512 last 1554943435 [Wed Apr 10 17:47:42 2019][2788580.905024] Lustre: Skipped 8 previous similar messages [Wed Apr 10 17:47:51 2019][2788590.709223] LustreError: 117911:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943581, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b796658e9c0/0x53d0f215b450c38c lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f215b450c3c4 expref: -99 pid: 117911 timeout: 0 lvb_type: 0 [Wed Apr 10 17:47:52 2019][2788591.467228] LustreError: 118278:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943582, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b560d670480/0x53d0f215b454195b lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 25 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118278 timeout: 0 lvb_type: 0 [Wed Apr 10 17:47:53 2019][2788592.822257] LustreError: 118552:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943583, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b5b0cd25580/0x53d0f215b45941ec lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f215b459421d expref: -99 pid: 118552 timeout: 0 lvb_type: 0 [Wed Apr 10 17:47:59 2019][2788598.491384] LustreError: 117869:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943589, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b55b7899440/0x53d0f215b46f0154 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f215b46f01af expref:reError: 117869:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 17:48:19 2019][2788617.900786] Lustre: 118481:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 17:48:19 2019][2788617.900786] req@ffff8b879cf0d100 x1630384201045264/t0(0) o36->809ab8a0-df79-48fa-443b-a45a215b260d@10.9.112.16@o2ib4:24/0 lens 520/2888 e 0 to 0 dl 1554943704 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:48:20 2019][2788619.434820] LustreError: 118272:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943610, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b6ce729ca40/0x53d0f215b4c49ee0 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f215b4c49eee expref: -99 pid: 118272 timeout: 0 lvb_type: 0 [Wed Apr 10 17:48:27 2019][2788626.735970] LustreError: 118376:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943617, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b6a5e750900/0x53d0f215b4d50e8f lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 28 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118376 timeout: 0 lvb_type: 0 [Wed Apr 10 17:52:01 2019][2788840.193532] Lustre: 118370:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 17:52:01 2019][2788840.193532] req@ffff8b6031fd3c00 x1630377444290208/t146141672457(0) o36->e4292209-c55b-cc00-a078-2d504f5a17aa@10.8.1.24@o2ib6:6/0 lens 536/2888 e 0 to 0 dl 1554943926 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 17:52:33 2019][2788872.202839] LustreError: 118746:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011190:0xa86:0x0] [Wed Apr 10 17:53:06 2019][2788905.284909] LustreError: 118266:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554943896, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b766267cec0/0x53d0f215ba23cc99 lrc: 3/1,0 mode: --/PR res: [0x2c00128f3:0xa9b0:0x0].0x0 bits 0x13/0x8 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118266 timeout: 0 lvb_type: 0 [Wed Apr 10 17:53:06 2019][2788905.324677] LustreError: 118266:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 17:54:56 2019][2789015.589240] LNet: Service thread pid 118266 was inactive for 200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 17:54:56 2019][2789015.606443] Pid: 118266, comm: mdt02_030 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 17:54:56 2019][2789015.616471] Call Trace: [Wed Apr 10 17:54:56 2019][2789015.619112] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 17:54:56 2019][2789015.626232] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 17:54:56 2019][2789015.633608] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 17:54:56 2019][2789015.640609] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 17:54:56 2019][2789015.647793] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 17:54:56 2019][2789015.654466] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 17:54:56 2019][2789015.661645] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 17:54:56 2019][2789015.668400] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 17:54:56 2019][2789015.675061] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 17:54:56 2019][2789015.682020] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 17:54:56 2019][2789015.689299] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 17:54:56 2019][2789015.695658] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 17:54:56 2019][2789015.702784] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 17:54:56 2019][2789015.710698] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 17:54:56 2019][2789015.717210] [] kthread+0xd1/0xe0 [Wed Apr 10 17:54:56 2019][2789015.722304] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 17:54:56 2019][2789015.728968] [] 0xffffffffffffffff [Wed Apr 10 17:54:56 2019][2789015.734179] LustreError: dumping log to /tmp/lustre-log.1554944096.118266 [Wed Apr 10 17:54:58 2019][2789017.120907] LNet: Service thread pid 118148 was inactive for 201.83s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 17:54:58 2019][2789017.138119] Pid: 118148, comm: mdt01_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 17:54:58 2019][2789017.148126] Call Trace: [Wed Apr 10 17:54:58 2019][2789017.150775] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 17:54:58 2019][2789017.157922] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 17:54:58 2019][2789017.165323] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 17:54:58 2019][2789017.172282] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 17:54:58 2019][2789017.178959] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 17:54:58 2019][2789017.185206] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 17:54:58 2019][2789017.191966] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 17:54:58 2019][2789017.197884] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 17:54:58 2019][2789017.205043] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 17:54:58 2019][2789017.212994] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 17:54:58 2019][2789017.219529] [] kthread+0xd1/0xe0 [Wed Apr 10 17:54:58 2019][2789017.224632] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 17:54:58 2019][2789017.231286] [] 0xffffffffffffffff [Wed Apr 10 17:56:56 2019][2789135.558598] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.22@o2ib4) [Wed Apr 10 17:56:56 2019][2789135.566096] Lustre: Skipped 1144 previous similar messages [Wed Apr 10 17:57:16 2019][2789155.234382] Lustre: fir-MDT0002: Client a132911f-e28d-2e04-1263-47066608ec3c (at 10.8.23.24@o2ib6) reconnecting [Wed Apr 10 17:57:16 2019][2789155.244668] Lustre: Skipped 1133 previous similar messages [Wed Apr 10 17:59:43 2019][2789301.938624] LustreError: 118845:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b8:0x45f4:0x0] [Wed Apr 10 17:59:50 2019][2789308.898405] Lustre: fir-MDT0000: haven't heard from client a5c8a438-2f61-6c1a-7178-b80e1ca94d85 (at 10.9.115.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4d0f4d2000, cur 1554944390 expire 1554944240 last 1554944163 [Wed Apr 10 17:59:50 2019][2789308.920306] Lustre: Skipped 5 previous similar messages [Wed Apr 10 18:06:57 2019][2789736.156211] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.18@o2ib6) [Wed Apr 10 18:06:57 2019][2789736.163533] Lustre: Skipped 1137 previous similar messages [Wed Apr 10 18:07:18 2019][2789757.197361] Lustre: fir-MDT0002: Client ee5f7d3d-8ffe-7ca2-e4f8-dd26c84b3cb6 (at 10.9.107.48@o2ib4) reconnecting [Wed Apr 10 18:07:18 2019][2789757.207715] Lustre: Skipped 1135 previous similar messages [Wed Apr 10 18:07:42 2019][2789781.357865] LustreError: 118845:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128f5:0xd85:0x0] [Wed Apr 10 18:16:59 2019][2790338.200281] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Wed Apr 10 18:16:59 2019][2790338.207609] Lustre: Skipped 1134 previous similar messages [Wed Apr 10 18:17:19 2019][2790358.340719] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Wed Apr 10 18:17:19 2019][2790358.351076] Lustre: Skipped 1128 previous similar messages [Wed Apr 10 18:24:25 2019][2790784.211012] LustreError: 118849:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010553:0x22ea:0x0] [Wed Apr 10 18:27:01 2019][2790940.555070] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Wed Apr 10 18:27:01 2019][2790940.562491] Lustre: Skipped 1138 previous similar messages [Wed Apr 10 18:27:19 2019][2790958.474534] Lustre: fir-MDT0002: Client a11a9980-d36a-715f-c190-a6765cde5927 (at 10.8.1.22@o2ib6) reconnecting [Wed Apr 10 18:27:19 2019][2790958.484717] Lustre: Skipped 1133 previous similar messages [Wed Apr 10 18:29:04 2019][2791062.932393] Lustre: 118253:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 18:29:04 2019][2791062.932393] req@ffff8b5c3c691200 x1630476198517648/t146149071982(0) o36->eda11444-0027-aea4-d226-92aced58871b@10.9.114.5@o2ib4:8/0 lens 504/2888 e 1 to 0 dl 1554946148 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 18:29:04 2019][2791062.962364] Lustre: 118253:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 18:29:13 2019][2791071.954585] Lustre: 118251:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 18:29:13 2019][2791071.954585] req@ffff8b59ef3df500 x1630386604024240/t0(0) o101->8edb63b6-d814-54d5-ddee-ddb62f3308b6@10.9.103.42@o2ib4:18/0 lens 576/3264 e 0 to 0 dl 1554946158 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 18:30:18 2019][2791137.896040] LustreError: 118167:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554946128, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b4e112ffbc0/0x53d0f215e2436d48 lrc: 3/1,0 mode: --/PR res: [0x2c00130a0:0x46b:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118167 timeout: 0 lvb_type: 0 [Wed Apr 10 18:30:19 2019][2791137.935718] LustreError: 118167:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 18:30:19 2019][2791138.766018] LustreError: 119056:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6e9a347200 x1628639149987280/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:25/0 lens 448/440 e 0 to 0 dl 1554946225 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 18:32:09 2019][2791247.956383] LNet: Service thread pid 118167 was inactive for 200.05s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 18:32:09 2019][2791247.973580] Pid: 118167, comm: mdt00_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 18:32:09 2019][2791247.983580] Call Trace: [Wed Apr 10 18:32:09 2019][2791247.986223] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 18:32:09 2019][2791247.993344] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 18:32:09 2019][2791248.000734] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 18:32:09 2019][2791248.007756] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 18:32:09 2019][2791248.014933] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 18:32:09 2019][2791248.021589] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 18:32:09 2019][2791248.028767] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 18:32:09 2019][2791248.035528] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 18:32:09 2019][2791248.042215] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 18:32:09 2019][2791248.049162] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 18:32:09 2019][2791248.056467] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 18:32:09 2019][2791248.062844] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 18:32:09 2019][2791248.069967] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 18:32:09 2019][2791248.077897] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 18:32:09 2019][2791248.084409] [] kthread+0xd1/0xe0 [Wed Apr 10 18:32:09 2019][2791248.089524] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 18:32:09 2019][2791248.096176] [] 0xffffffffffffffff [Wed Apr 10 18:32:09 2019][2791248.101379] LustreError: dumping log to /tmp/lustre-log.1554946329.118167 [Wed Apr 10 18:32:10 2019][2791249.429813] LNet: Service thread pid 117560 was inactive for 201.53s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 18:32:10 2019][2791249.447038] Pid: 117560, comm: mdt00_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 18:32:10 2019][2791249.457052] Call Trace: [Wed Apr 10 18:32:10 2019][2791249.459695] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 18:32:10 2019][2791249.466816] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 18:32:10 2019][2791249.474209] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 18:32:10 2019][2791249.481586] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 18:32:10 2019][2791249.489582] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 18:32:10 2019][2791249.496191] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 18:32:10 2019][2791249.502931] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 18:32:10 2019][2791249.508808] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 18:32:10 2019][2791249.515967] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 18:32:10 2019][2791249.523868] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 18:32:10 2019][2791249.530389] [] kthread+0xd1/0xe0 [Wed Apr 10 18:32:10 2019][2791249.535497] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 18:32:10 2019][2791249.542166] [] 0xffffffffffffffff [Wed Apr 10 18:32:21 2019][2791259.942198] Lustre: fir-MDT0002: haven't heard from client b2ab8d48-0658-141b-0b32-f738fa655f05 (at 10.9.105.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b77957ebc00, cur 1554946341 expire 1554946191 last 1554946114 [Wed Apr 10 18:32:21 2019][2791259.964188] Lustre: Skipped 2 previous similar messages [Wed Apr 10 18:36:11 2019][2791489.944654] Lustre: fir-MDT0000: haven't heard from client f6f4af16-1479-93f2-5af6-1e2c846a1ab6 (at 10.9.112.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4cc3937c00, cur 1554946571 expire 1554946421 last 1554946344 [Wed Apr 10 18:36:11 2019][2791489.966534] Lustre: Skipped 23 previous similar messages [Wed Apr 10 18:37:03 2019][2791542.617789] Lustre: fir-MDT0002: Connection restored to (at 10.9.107.48@o2ib4) [Wed Apr 10 18:37:03 2019][2791542.625282] Lustre: Skipped 1173 previous similar messages [Wed Apr 10 18:37:19 2019][2791558.537974] Lustre: fir-MDT0002: Client 3c1359c4-6f8a-28e8-3fe1-a0a63b3ad896 (at 10.9.102.1@o2ib4) reconnecting [Wed Apr 10 18:37:19 2019][2791558.548232] Lustre: Skipped 1163 previous similar messages [Wed Apr 10 18:39:02 2019][2791661.664886] LustreError: 118746:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b6a56b4ec50 x1629295260914544/t0(0) o4->56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5@10.8.8.8@o2ib6:8/0 lens 488/448 e 0 to 0 dl 1554946748 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 18:39:02 2019][2791661.688944] Lustre: fir-MDT0002: Bulk IO write error with 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6), client will retry: rc = -110 [Wed Apr 10 18:44:33 2019][2791992.728751] LustreError: 118619:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010783:0x2282:0x0] [Wed Apr 10 18:44:40 2019][2791998.945383] LNetError: 117130:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Wed Apr 10 18:45:24 2019][2792043.292941] LustreError: 118853:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012962:0x685:0x0] [Wed Apr 10 18:47:05 2019][2792144.765236] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Wed Apr 10 18:47:05 2019][2792144.775600] Lustre: Skipped 1175 previous similar messages [Wed Apr 10 18:47:19 2019][2792158.789854] Lustre: fir-MDT0002: Client a132911f-e28d-2e04-1263-47066608ec3c (at 10.8.23.24@o2ib6) reconnecting [Wed Apr 10 18:47:19 2019][2792158.800125] Lustre: Skipped 1176 previous similar messages [Wed Apr 10 18:57:06 2019][2792745.141124] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.26@o2ib4) [Wed Apr 10 18:57:06 2019][2792745.148626] Lustre: Skipped 1195 previous similar messages [Wed Apr 10 18:57:21 2019][2792760.917812] Lustre: fir-MDT0002: Client ee5f7d3d-8ffe-7ca2-e4f8-dd26c84b3cb6 (at 10.9.107.48@o2ib4) reconnecting [Wed Apr 10 18:57:22 2019][2792760.928168] Lustre: Skipped 1184 previous similar messages [Wed Apr 10 18:57:57 2019][2792796.329404] Lustre: 118547:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 18:57:57 2019][2792796.329404] req@ffff8b501734bf00 x1628648933336992/t146156192457(0) o36->1bde1f20-37f1-ed48-746c-38acfc0e9a71@10.9.106.55@o2ib4:2/0 lens 512/2888 e 1 to 0 dl 1554947882 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 18:58:20 2019][2792819.007633] LustreError: 119155:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b7625e3bf00 x1628639155413872/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:26/0 lens 448/440 e 0 to 0 dl 1554947906 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 18:59:12 2019][2792871.302987] LustreError: 117463:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554947862, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b55a46f6780/0x53d0f216021f1caa lrc: 3/1,0 mode: --/PR res: [0x2c0010804:0x96e3:0x0].0x0 bits 0x13/0x8 rrc: 17 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 117463 timeout: 0 lvb_type: 0 [Wed Apr 10 18:59:12 2019][2792871.342828] LustreError: 117463:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 19:01:02 2019][2792981.625339] LNet: Service thread pid 117463 was inactive for 200.31s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 19:01:02 2019][2792981.642533] Pid: 117463, comm: mdt00_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 19:01:02 2019][2792981.652535] Call Trace: [Wed Apr 10 19:01:02 2019][2792981.655177] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 19:01:02 2019][2792981.662317] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 19:01:02 2019][2792981.669710] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 19:01:02 2019][2792981.676730] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 19:01:02 2019][2792981.683905] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 19:01:02 2019][2792981.690562] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 19:01:02 2019][2792981.697746] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 19:01:02 2019][2792981.704516] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 19:01:02 2019][2792981.711201] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 19:01:02 2019][2792981.718155] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 19:01:02 2019][2792981.725435] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 19:01:02 2019][2792981.731791] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 19:01:02 2019][2792981.738905] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 19:01:02 2019][2792981.746975] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 19:01:02 2019][2792981.753500] [] kthread+0xd1/0xe0 [Wed Apr 10 19:01:02 2019][2792981.758608] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 19:01:02 2019][2792981.765260] [] 0xffffffffffffffff [Wed Apr 10 19:01:02 2019][2792981.770478] LustreError: dumping log to /tmp/lustre-log.1554948062.117463 [Wed Apr 10 19:01:04 2019][2792983.085324] LNet: Service thread pid 118260 was inactive for 201.78s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 19:01:04 2019][2792983.102527] Pid: 118260, comm: mdt00_043 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 19:01:04 2019][2792983.112554] Call Trace: [Wed Apr 10 19:01:04 2019][2792983.115205] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 19:01:04 2019][2792983.122338] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 19:01:04 2019][2792983.129719] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 19:01:04 2019][2792983.136636] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 19:01:04 2019][2792983.143297] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 19:01:04 2019][2792983.149529] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 19:01:04 2019][2792983.156309] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 19:01:04 2019][2792983.162188] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 19:01:04 2019][2792983.169322] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 19:01:04 2019][2792983.177259] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 19:01:04 2019][2792983.183768] [] kthread+0xd1/0xe0 [Wed Apr 10 19:01:04 2019][2792983.188871] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 19:01:04 2019][2792983.195587] [] 0xffffffffffffffff [Wed Apr 10 19:01:33 2019][2793012.654006] Lustre: 118340:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 19:01:33 2019][2793012.654006] req@ffff8b6ea9bff800 x1628642465741440/t146157121423(0) o36->6d84db07-946f-3a55-fea2-55904c526e1b@10.8.1.21@o2ib6:8/0 lens 544/2888 e 1 to 0 dl 1554948098 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 19:01:33 2019][2793012.683858] Lustre: 118340:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Apr 10 19:02:48 2019][2793087.617602] LustreError: 118457:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554948078, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b882fbdf500/0x53d0f216076665e4 lrc: 3/1,0 mode: --/PR res: [0x2c0010636:0x5e77:0x0].0x0 bits 0x13/0x8 rrc: 12 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118457 timeout: 0 lvb_type: 0 [Wed Apr 10 19:02:48 2019][2793087.657439] LustreError: 118457:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Wed Apr 10 19:04:38 2019][2793197.693985] LNet: Service thread pid 118275 was inactive for 200.06s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 19:04:38 2019][2793197.711187] Pid: 118275, comm: mdt01_048 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 19:04:38 2019][2793197.721181] Call Trace: [Wed Apr 10 19:04:38 2019][2793197.723819] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 19:04:38 2019][2793197.730931] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 19:04:38 2019][2793197.738314] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 19:04:38 2019][2793197.745317] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 19:04:38 2019][2793197.752493] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Wed Apr 10 19:04:38 2019][2793197.759669] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 19:04:38 2019][2793197.766412] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 19:04:38 2019][2793197.773068] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 19:04:38 2019][2793197.780002] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 19:04:38 2019][2793197.787282] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 19:04:38 2019][2793197.793617] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 19:04:38 2019][2793197.800733] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 19:04:38 2019][2793197.808619] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 19:04:38 2019][2793197.815140] [] kthread+0xd1/0xe0 [Wed Apr 10 19:04:38 2019][2793197.820232] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 19:04:38 2019][2793197.826881] [] 0xffffffffffffffff [Wed Apr 10 19:04:38 2019][2793197.832074] LustreError: dumping log to /tmp/lustre-log.1554948278.118275 [Wed Apr 10 19:04:39 2019][2793198.473785] Pid: 118457, comm: mdt03_045 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 19:04:39 2019][2793198.483783] Call Trace: [Wed Apr 10 19:04:39 2019][2793198.486424] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.493553] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.500919] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 19:04:39 2019][2793198.507923] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 19:04:39 2019][2793198.515114] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 19:04:39 2019][2793198.521774] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 19:04:39 2019][2793198.528966] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 19:04:39 2019][2793198.535711] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 19:04:39 2019][2793198.542383] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.549325] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.556605] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.562949] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.570109] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.578005] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.584505] [] kthread+0xd1/0xe0 [Wed Apr 10 19:04:39 2019][2793198.589590] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 19:04:39 2019][2793198.596248] [] 0xffffffffffffffff [Wed Apr 10 19:04:39 2019][2793198.601444] Pid: 118402, comm: mdt01_102 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 19:04:39 2019][2793198.611443] Call Trace: [Wed Apr 10 19:04:39 2019][2793198.614082] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.621205] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.628576] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 19:04:39 2019][2793198.635501] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 19:04:39 2019][2793198.642159] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 19:04:39 2019][2793198.648390] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 19:04:39 2019][2793198.655133] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 19:04:39 2019][2793198.661047] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.668165] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.676095] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 19:04:39 2019][2793198.682594] [] kthread+0xd1/0xe0 [Wed Apr 10 19:04:39 2019][2793198.687683] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 19:04:39 2019][2793198.694331] [] 0xffffffffffffffff [Wed Apr 10 19:07:06 2019][2793345.319625] Lustre: fir-MDT0002: Connection restored to 3dde01db-8bab-e6a5-6af0-a17c56d94df2 (at 10.9.101.66@o2ib4) [Wed Apr 10 19:07:06 2019][2793345.330237] Lustre: Skipped 1301 previous similar messages [Wed Apr 10 19:07:22 2019][2793361.026691] Lustre: fir-MDT0002: Client 563d53a1-cc0d-a13f-4381-648a7da03caa (at 10.9.105.60@o2ib4) reconnecting [Wed Apr 10 19:07:22 2019][2793361.037053] Lustre: Skipped 1281 previous similar messages [Wed Apr 10 19:12:01 2019][2793640.338567] Lustre: 118368:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 19:12:01 2019][2793640.338567] req@ffff8b6b9df52d00 x1630383994091296/t0(0) o101->ad81974e-6a7d-6ab2-8498-67252e6b5d22@10.9.113.1@o2ib4:6/0 lens 480/568 e 1 to 0 dl 1554948726 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 19:12:01 2019][2793640.367557] Lustre: 118368:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 19:12:15 2019][2793654.656910] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.27.27@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8b6b4ef5cec0/0x53d0f216125a3e83 lrc: 3/0,0 mode: PW/PW res: [0x20000fcb7:0xad77:0x0].0x0 bits 0x40/0x0 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.8.27.27@o2ib6 remote: 0x51dc737db69348f3 expref: 941 pid: 118225 timeout: 2793591 lvb_type: 0 [Wed Apr 10 19:12:41 2019][2793679.991481] Lustre: fir-MDT0000: haven't heard from client 9fb7f2d4-f4f7-ebda-25d8-184ef751d76f (at 10.9.113.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5842772800, cur 1554948761 expire 1554948611 last 1554948534 [Wed Apr 10 19:12:41 2019][2793680.013457] Lustre: Skipped 2 previous similar messages [Wed Apr 10 19:12:45 2019][2793684.267742] LustreError: 118599:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001082f:0x203e:0x0] [Wed Apr 10 19:15:27 2019][2793845.999754] Lustre: MGS: haven't heard from client b452493f-dcad-f47f-559b-e54fd147afb0 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7550b66000, cur 1554948927 expire 1554948777 last 1554948700 [Wed Apr 10 19:15:27 2019][2793846.020873] Lustre: Skipped 2 previous similar messages [Wed Apr 10 19:17:06 2019][2793945.398899] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.22@o2ib4) [Wed Apr 10 19:17:06 2019][2793945.406385] Lustre: Skipped 1321 previous similar messages [Wed Apr 10 19:17:23 2019][2793962.319258] Lustre: fir-MDT0002: Client e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6) reconnecting [Wed Apr 10 19:17:23 2019][2793962.329440] Lustre: Skipped 1320 previous similar messages [Wed Apr 10 19:24:52 2019][2794411.007579] Lustre: fir-MDT0000: haven't heard from client 685c7782-a6d3-9ae9-f6a1-6708a0b985ae (at 10.9.114.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4d11653c00, cur 1554949492 expire 1554949342 last 1554949265 [Wed Apr 10 19:24:52 2019][2794411.029544] Lustre: Skipped 3 previous similar messages [Wed Apr 10 19:27:06 2019][2794545.885136] Lustre: fir-MDT0002: Connection restored to 48747558-7da6-0440-a86c-cce04c28222a (at 10.8.7.22@o2ib6) [Wed Apr 10 19:27:06 2019][2794545.895575] Lustre: Skipped 1326 previous similar messages [Wed Apr 10 19:27:23 2019][2794562.731655] Lustre: fir-MDT0002: Client 99b2b05a-80e4-edd2-5271-b136b8e4528d (at 10.9.108.67@o2ib4) reconnecting [Wed Apr 10 19:27:23 2019][2794562.742006] Lustre: Skipped 1318 previous similar messages [Wed Apr 10 19:28:07 2019][2794606.470814] LustreError: 118898:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011259:0xdcc:0x0] [Wed Apr 10 19:28:17 2019][2794616.770382] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 19:28:17 2019][2794616.787781] LustreError: Skipped 2 previous similar messages [Wed Apr 10 19:28:26 2019][2794625.210258] LustreError: 118852:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128b7:0x50e:0x0] [Wed Apr 10 19:28:35 2019][2794634.012451] Lustre: fir-MDT0000: haven't heard from client 8bee312c-adaa-4011-4be3-7f8f8eac98cb (at 10.9.101.56@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5c8c778c00, cur 1554949715 expire 1554949565 last 1554949488 [Wed Apr 10 19:28:35 2019][2794634.034420] Lustre: Skipped 2 previous similar messages [Wed Apr 10 19:28:42 2019][2794641.858853] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 19:28:42 2019][2794641.869115] Lustre: Skipped 1 previous similar message [Wed Apr 10 19:29:03 2019][2794662.126376] LustreError: 118767:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011190:0xde6:0x0] [Wed Apr 10 19:30:18 2019][2794737.796908] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 19:30:18 2019][2794737.796975] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 19:31:36 2019][2794815.016471] Lustre: fir-MDT0000: haven't heard from client 056b5f54-6fde-d24e-91db-e32a1e5acf9e (at 10.9.101.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5842771400, cur 1554949896 expire 1554949746 last 1554949669 [Wed Apr 10 19:31:36 2019][2794815.038440] Lustre: Skipped 15 previous similar messages [Wed Apr 10 19:32:23 2019][2794862.407801] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 19:32:23 2019][2794862.425200] LustreError: Skipped 2 previous similar messages [Wed Apr 10 19:32:48 2019][2794887.496100] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 19:37:09 2019][2795148.142205] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Wed Apr 10 19:37:09 2019][2795148.152554] Lustre: Skipped 1328 previous similar messages [Wed Apr 10 19:37:19 2019][2795158.039508] Lustre: MGS: haven't heard from client b452493f-dcad-f47f-559b-e54fd147afb0 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6fc039c000, cur 1554950239 expire 1554950089 last 1554950012 [Wed Apr 10 19:37:19 2019][2795158.060630] Lustre: Skipped 19 previous similar messages [Wed Apr 10 19:37:24 2019][2795163.061762] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 19:37:24 2019][2795163.072286] Lustre: Skipped 1327 previous similar messages [Wed Apr 10 19:40:12 2019][2795331.502788] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 19:42:12 2019][2795451.039438] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 19:44:19 2019][2795578.774567] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 19:44:20 2019][2795579.162979] LustreError: 117297:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b8ad02b6c50 x1629181947120320/t0(0) o256->b452493f-dcad-f47f-559b-e54fd147afb0@10.8.0.65@o2ib6:26/0 lens 304/240 e 0 to 0 dl 1554950666 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 19:44:37 2019][2795596.279155] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.113.13@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 19:44:37 2019][2795596.296715] LustreError: Skipped 3 previous similar messages [Wed Apr 10 19:47:09 2019][2795748.389520] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.60@o2ib4) [Wed Apr 10 19:47:09 2019][2795748.397025] Lustre: Skipped 1337 previous similar messages [Wed Apr 10 19:47:24 2019][2795763.393380] Lustre: fir-MDT0002: Client 1bde1f20-37f1-ed48-746c-38acfc0e9a71 (at 10.9.106.55@o2ib4) reconnecting [Wed Apr 10 19:47:24 2019][2795763.403948] Lustre: Skipped 1331 previous similar messages [Wed Apr 10 19:49:10 2019][2795869.164791] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.114.14@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 19:49:10 2019][2795869.182336] LustreError: Skipped 7 previous similar messages [Wed Apr 10 19:49:21 2019][2795880.731866] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 19:49:31 2019][2795890.039626] Lustre: fir-MDT0002: haven't heard from client 39b21bb1-06bc-8c69-c9c3-5986c42b070c (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6e5a6e6000, cur 1554950971 expire 1554950821 last 1554950744 [Wed Apr 10 19:49:31 2019][2795890.061413] Lustre: Skipped 4 previous similar messages [Wed Apr 10 19:55:54 2019][2796273.405854] Lustre: MGS: Received new LWP connection from 10.8.0.65@o2ib6, removing former export from same NID [Wed Apr 10 19:56:26 2019][2796305.549272] LustreError: 119139:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b7cb6336050 x1628639164488224/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:2/0 lens 448/440 e 0 to 0 dl 1554951392 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 19:57:09 2019][2796348.413203] Lustre: fir-MDT0002: Connection restored to 88ba37a5-db4c-fc43-b4b3-806ad9c864f1 (at 10.9.102.2@o2ib4) [Wed Apr 10 19:57:09 2019][2796348.423732] Lustre: Skipped 1335 previous similar messages [Wed Apr 10 19:57:24 2019][2796363.408912] Lustre: fir-MDT0002: Client f4b95b51-aa49-8c42-0e74-2b9d33fd9533 (at 10.9.101.64@o2ib4) reconnecting [Wed Apr 10 19:57:24 2019][2796363.419268] Lustre: Skipped 1322 previous similar messages [Wed Apr 10 19:57:49 2019][2796388.132428] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.101.56@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 19:57:49 2019][2796388.149973] LustreError: Skipped 26 previous similar messages [Wed Apr 10 20:01:33 2019][2796612.054745] Lustre: fir-MDT0000: haven't heard from client b12115fe-7ea4-fe17-7e60-7528a752fb31 (at 10.9.114.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4ce9612c00, cur 1554951693 expire 1554951543 last 1554951466 [Wed Apr 10 20:01:33 2019][2796612.076715] Lustre: Skipped 3 previous similar messages [Wed Apr 10 20:02:45 2019][2796684.151260] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554951758/real 1554951758] req@ffff8b6f35fb6300 x1630439158022992/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554951765 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 20:02:52 2019][2796691.178411] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554951765/real 1554951765] req@ffff8b6f35fb6300 x1630439158022992/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554951772 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 20:02:53 2019][2796692.425429] Lustre: 117532:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 20:02:53 2019][2796692.425429] req@ffff8b781b791200 x1630377137128224/t0(0) o36->27c28ffe-2196-b1ed-69fc-d13ef84dbe48@10.9.114.1@o2ib4:28/0 lens 512/2888 e 1 to 0 dl 1554951778 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:02:59 2019][2796698.205560] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554951772/real 1554951772] req@ffff8b6f35fb6300 x1630439158022992/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554951779 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 20:03:03 2019][2796702.815643] Lustre: 118573:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 20:03:03 2019][2796702.815643] req@ffff8b5903282400 x1629295525798384/t0(0) o101->44c9d3f9-8f6e-58fb-b793-b3c99329a24e@10.9.108.32@o2ib4:8/0 lens 576/3264 e 1 to 0 dl 1554951788 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:03:13 2019][2796712.232853] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554951786/real 1554951786] req@ffff8b6f35fb6300 x1630439158022992/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554951793 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 20:03:13 2019][2796712.260223] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Wed Apr 10 20:03:15 2019][2796714.477892] Lustre: 117532:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 20:03:15 2019][2796714.477892] req@ffff8b723c634800 x1628593316636464/t0(0) o101->b888ab53-f095-b22e-cdd5-46beb027c82e@10.8.17.5@o2ib6:20/0 lens 1776/3288 e 0 to 0 dl 1554951800 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:03:34 2019][2796733.270298] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554951807/real 1554951807] req@ffff8b6f35fb6300 x1630439158022992/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554951814 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 20:03:34 2019][2796733.297641] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Wed Apr 10 20:03:58 2019][2796757.392794] Lustre: 118363:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 20:03:58 2019][2796757.392794] req@ffff8b5c8fe3d700 x1629298555203568/t0(0) o101->ee6e221c-c474-4552-5653-54cec6cd0dcd@10.9.108.23@o2ib4:3/0 lens 1776/3288 e 1 to 0 dl 1554951843 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:04:06 2019][2796765.414959] Lustre: 117466:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 20:04:06 2019][2796765.414959] req@ffff8b6ce5d10f00 x1629035064872592/t0(0) o101->36e2649c-5bde-7360-b2cc-505510b79efe@10.8.7.26@o2ib6:11/0 lens 1776/3288 e 0 to 0 dl 1554951851 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:04:09 2019][2796768.309030] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554951842/real 1554951842] req@ffff8b6f35fb6300 x1630439158022992/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1554951849 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Wed Apr 10 20:04:09 2019][2796768.336386] Lustre: 117471:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Wed Apr 10 20:04:18 2019][2796777.779220] LustreError: 118394:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554951768, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b5a95c09f80/0x53d0f2162f6abb7b lrc: 3/1,0 mode: --/PR res: [0x2c001306c:0x6d03:0x0].0x0 bits 0x13/0x0 rrc: 171 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118394 timeout: 0 lvb_type: 0 [Wed Apr 10 20:04:21 2019][2796780.051268] LustreError: 118576:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554951770, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b7412a49200/0x53d0f2162f6f59c5 lrc: 3/0,1 mode: --/CW res: [0x2c001306c:0x6d03:0x0].0x0 bits 0x2/0x0 rrc: 173 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118576 timeout: 0 lvb_type: 0 [Wed Apr 10 20:04:33 2019][2796792.517527] Lustre: 118437:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 20:04:33 2019][2796792.517527] req@ffff8b7717bd9b00 x1629293582177888/t0(0) o101->3d887935-8bfc-5320-4aa6-6c952f9fe8d7@10.8.8.11@o2ib6:8/0 lens 1776/3288 e 0 to 0 dl 1554951878 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:05:05 2019][2796824.880209] Lustre: 118321:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 20:05:05 2019][2796824.880209] req@ffff8b60003ff500 x1629293677806912/t0(0) o101->930cf174-e8e0-277a-21bd-b711955e5992@10.8.1.19@o2ib6:10/0 lens 1776/3288 e 0 to 0 dl 1554951910 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:05:05 2019][2796824.909464] Lustre: 118321:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Apr 10 20:05:11 2019][2796830.349329] LustreError: 118310:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554951821, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b85b1f82400/0x53d0f2162fedba28 lrc: 3/0,1 mode: --/CW res: [0x2c001306c:0x6d03:0x0].0x0 bits 0x2/0x0 rrc: 175 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118310 timeout: 0 lvb_type: 0 [Wed Apr 10 20:05:12 2019][2796831.348376] LustreError: 117471:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.8@o2ib6) failed to reply to blocking AST (req@ffff8b6f35fb6300 x1630439158022992 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b55aa37a400/0x53d0f21627dd081d lrc: 4/0,0 mode: PR/PR res: [0x2c00128ee:0x8566:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.9.8@o2ib6 remote: 0x1d4149bfb6d39dba expref: 4997 pid: 118547 timeout: 2796910 lvb_type: 0 [Wed Apr 10 20:05:12 2019][2796831.391336] LustreError: 138-a: fir-MDT0002: A client on nid 10.8.9.8@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [Wed Apr 10 20:05:12 2019][2796831.403873] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.9.8@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8b55aa37a400/0x53d0f21627dd081d lrc: 3/0,0 mode: PR/PR res: [0x2c00128ee:0x8566:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.9.8@o2ib6 remote: 0x1d4149bfb6d39dba expref: 4998 pid: 118547 timeout: 0 lvb_type: 0 [Wed Apr 10 20:05:12 2019][2796831.510923] LustreError: 118554:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b5c3c694200 x1630439159074144/t0(0) o104->fir-MDT0002@10.8.9.8@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Wed Apr 10 20:05:12 2019][2796831.532036] LustreError: 118554:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [Wed Apr 10 20:06:56 2019][2796935.643449] LustreError: 119180:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b7432fc6f00 x1628639166336832/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:2/0 lens 448/440 e 0 to 0 dl 1554952022 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:07:09 2019][2796948.919681] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.22@o2ib4) [Wed Apr 10 20:07:09 2019][2796948.927173] Lustre: Skipped 1379 previous similar messages [Wed Apr 10 20:07:28 2019][2796967.102975] Lustre: fir-MDT0002: Client f5140abb-0b33-a9a1-9459-14bf253ba61e (at 10.9.106.60@o2ib4) reconnecting [Wed Apr 10 20:07:28 2019][2796967.113325] Lustre: Skipped 1350 previous similar messages [Wed Apr 10 20:07:49 2019][2796988.635673] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.26.2@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 20:07:49 2019][2796988.653043] LustreError: Skipped 134 previous similar messages [Wed Apr 10 20:10:05 2019][2797124.647562] LustreError: 119112:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b648ebfcb00 x1628639166743456/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:11/0 lens 448/440 e 0 to 0 dl 1554952211 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:17:09 2019][2797548.995664] Lustre: fir-MDT0002: Connection restored to (at 10.9.107.48@o2ib4) [Wed Apr 10 20:17:09 2019][2797549.003153] Lustre: Skipped 1323 previous similar messages [Wed Apr 10 20:17:28 2019][2797567.259542] Lustre: fir-MDT0002: Client 931a3723-3247-0feb-2066-0d2fae060ca7 (at 10.9.101.66@o2ib4) reconnecting [Wed Apr 10 20:17:28 2019][2797567.269893] Lustre: Skipped 1324 previous similar messages [Wed Apr 10 20:17:50 2019][2797589.736450] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.26.2@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 20:17:50 2019][2797589.753823] LustreError: Skipped 179 previous similar messages [Wed Apr 10 20:19:32 2019][2797691.724885] LustreError: 119249:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b722c2b7200 x1628639168337328/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:8/0 lens 448/440 e 0 to 0 dl 1554952778 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:27:12 2019][2798151.076707] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Wed Apr 10 20:27:12 2019][2798151.084020] Lustre: Skipped 1324 previous similar messages [Wed Apr 10 20:27:29 2019][2798168.479858] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Wed Apr 10 20:27:29 2019][2798168.490211] Lustre: Skipped 1317 previous similar messages [Wed Apr 10 20:27:51 2019][2798190.837316] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.26.2@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 20:27:51 2019][2798190.854684] LustreError: Skipped 184 previous similar messages [Wed Apr 10 20:28:17 2019][2798216.800277] LustreError: 119217:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8c4a3bb600 x1628639169568384/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:23/0 lens 448/440 e 0 to 0 dl 1554953303 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:37:13 2019][2798752.436366] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Wed Apr 10 20:37:13 2019][2798752.443682] Lustre: Skipped 1321 previous similar messages [Wed Apr 10 20:37:29 2019][2798768.722886] Lustre: fir-MDT0002: Client f53d399b-a6b5-36d0-034f-8c8c0efbb65f (at 10.9.106.11@o2ib4) reconnecting [Wed Apr 10 20:37:29 2019][2798768.733239] Lustre: Skipped 1324 previous similar messages [Wed Apr 10 20:37:52 2019][2798791.938155] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.26.2@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 20:37:52 2019][2798791.955545] LustreError: Skipped 191 previous similar messages [Wed Apr 10 20:38:57 2019][2798856.237141] LustreError: 118772:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001083f:0xcd0e:0x0] [Wed Apr 10 20:38:57 2019][2798856.237177] LustreError: 118597:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001083f:0xcd0f:0x0] [Wed Apr 10 20:41:14 2019][2798993.884690] LustreError: 119085:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6418098000 x1628639171634752/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:20/0 lens 448/440 e 0 to 0 dl 1554954080 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:46:08 2019][2799287.953898] LustreError: 119160:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b7a25399e00 x1628639172178752/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:14/0 lens 448/440 e 0 to 0 dl 1554954374 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:47:11 2019][2799350.957128] LustreError: 117902:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b68f629f050 x1628639172293360/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:17/0 lens 448/440 e 0 to 0 dl 1554954437 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:47:13 2019][2799352.614380] Lustre: fir-MDT0002: Connection restored to (at 10.8.26.21@o2ib6) [Wed Apr 10 20:47:13 2019][2799352.621784] Lustre: Skipped 1323 previous similar messages [Wed Apr 10 20:47:30 2019][2799369.553325] Lustre: fir-MDT0002: Client 44691687-d1db-8fd7-cfeb-5cbdecfc902b (at 10.9.102.26@o2ib4) reconnecting [Wed Apr 10 20:47:30 2019][2799369.563674] Lustre: Skipped 1313 previous similar messages [Wed Apr 10 20:47:53 2019][2799393.039103] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.8.26.2@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 20:47:53 2019][2799393.056471] LustreError: Skipped 191 previous similar messages [Wed Apr 10 20:49:59 2019][2799518.964101] LustreError: 118531:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8a887d3050 x1628639172605312/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:5/0 lens 448/440 e 0 to 0 dl 1554954605 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:51:44 2019][2799623.586593] Lustre: DEBUG MARKER: Wed Apr 10 20:51:44 2019 [Wed Apr 10 20:51:44 2019][2799623.586593] [Wed Apr 10 20:53:12 2019][2799711.247751] Lustre: 117178:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554954785/real 1554954785] req@ffff8b83eba22400 x1630439181131200/t0(0) o41->fir-MDT0003-osp-MDT0002@10.0.10.52@o2ib7:24/4 lens 224/368 e 0 to 1 dl 1554954792 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Wed Apr 10 20:53:12 2019][2799711.276150] Lustre: 117178:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [Wed Apr 10 20:53:12 2019][2799711.286162] Lustre: fir-MDT0003-osp-MDT0002: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Apr 10 20:53:12 2019][2799711.990769] Lustre: fir-MDT0001-osp-MDT0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete [Wed Apr 10 20:53:12 2019][2799712.006927] Lustre: Skipped 1 previous similar message [Wed Apr 10 20:53:19 2019][2799718.589897] LNetError: 117113:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds [Wed Apr 10 20:53:19 2019][2799718.600160] LNetError: 117113:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (6): c: 0, oc: 0, rc: 8 [Wed Apr 10 20:54:08 2019][2799767.589949] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds [Wed Apr 10 20:55:00 2019][2799819.592060] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 1 seconds [Wed Apr 10 20:55:00 2019][2799819.602405] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Apr 10 20:55:45 2019][2799864.592040] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds [Wed Apr 10 20:55:45 2019][2799864.602385] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Apr 10 20:55:49 2019][2799868.593123] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds [Wed Apr 10 20:56:39 2019][2799918.593211] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds [Wed Apr 10 20:56:39 2019][2799918.603558] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 2 previous similar messages [Wed Apr 10 20:56:42 2019][2799921.814281] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 20:56:42 2019][2799921.814281] req@ffff8b72ba22fb00 x1630383819170656/t0(0) o36->83f54ca4-beb5-b0c8-46fd-04b8cce1d8ed@10.8.29.2@o2ib6:17/0 lens 536/2888 e 1 to 0 dl 1554955007 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:56:44 2019][2799923.126497] Lustre: MGS: haven't heard from client 6e66d1be-66d9-7058-c001-00c06a3f1758 (at 10.0.10.52@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b523f35d800, cur 1554955004 expire 1554954854 last 1554954777 [Wed Apr 10 20:56:44 2019][2799923.147727] Lustre: Skipped 4 previous similar messages [Wed Apr 10 20:56:47 2019][2799926.827383] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 20:56:47 2019][2799926.827383] req@ffff8b5f80bf3900 x1628732713996448/t0(0) o36->8432bfa2-9d59-805c-f710-2f7b2a2f50e9@10.9.106.62@o2ib4:22/0 lens 600/2888 e 1 to 0 dl 1554955012 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:56:47 2019][2799926.856558] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 20:57:02 2019][2799941.128701] Lustre: 118262:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 20:57:02 2019][2799941.128701] req@ffff8b7cbce02a00 x1630440981249568/t0(0) o36->e75095d8-7c9e-d169-de21-b278ca100ccd@10.9.101.10@o2ib4:7/0 lens 776/2888 e 0 to 0 dl 1554955027 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:57:14 2019][2799953.191360] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.35@o2ib4) [Wed Apr 10 20:57:14 2019][2799953.198852] Lustre: Skipped 1322 previous similar messages [Wed Apr 10 20:57:19 2019][2799958.175058] Lustre: 118171:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 20:57:19 2019][2799958.175058] req@ffff8b72fc2b1e00 x1628657839872800/t0(0) o36->99215ebf-6ff4-c562-22f3-08c5a008959e@10.9.106.59@o2ib4:24/0 lens 600/2888 e 1 to 0 dl 1554955044 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:57:19 2019][2799958.204239] Lustre: 118171:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Wed Apr 10 20:57:29 2019][2799968.594271] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds [Wed Apr 10 20:57:29 2019][2799968.604618] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Apr 10 20:57:30 2019][2799969.860157] Lustre: fir-MDT0000: Client 83f54ca4-beb5-b0c8-46fd-04b8cce1d8ed (at 10.8.29.2@o2ib6) reconnecting [Wed Apr 10 20:57:30 2019][2799969.870331] Lustre: Skipped 1331 previous similar messages [Wed Apr 10 20:57:53 2019][2799993.071835] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.107.60@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 20:57:53 2019][2799993.089403] LustreError: Skipped 12338 previous similar messages [Wed Apr 10 20:57:58 2019][2799997.976911] LustreError: 118321:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554954988, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b603ea79b00/0x53d0f21648b0ee29 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 14 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118321 timeout: 0 lvb_type: 0 [Wed Apr 10 20:58:02 2019][2800002.071001] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Apr 10 20:58:02 2019][2800002.086141] Lustre: Skipped 1 previous similar message [Wed Apr 10 20:58:02 2019][2800002.091478] LustreError: 118393:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554954992, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b69ecf6ba80/0x53d0f21648b8df01 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f21648b8df08 expref: -99 pid: 118393 timeout: 0 lvb_type: 0 [Wed Apr 10 20:58:03 2019][2800002.131841] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Apr 10 20:58:07 2019][2800006.161097] LustreError: 118364:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554954997, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b7a863d4140/0x53d0f21648c10c7f lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 16 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118364 timeout: 0 lvb_type: 0 [Wed Apr 10 20:58:19 2019][2800018.874370] Lustre: 118316:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 20:58:19 2019][2800018.874370] req@ffff8b83f8ea6300 x1628546345860640/t0(0) o36->b0387204-7ccc-80ac-5f41-f65a8a09bdb3@10.9.106.71@o2ib4:24/0 lens 600/2888 e 0 to 0 dl 1554955104 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:58:19 2019][2800018.903635] Lustre: 118316:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Apr 10 20:58:22 2019][2800021.900435] LustreError: 118541:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955012, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b7ba0bb1f80/0x53d0f21648dd578e lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 17 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118541 timeout: 0 lvb_type: 0 [Wed Apr 10 20:58:33 2019][2800032.595663] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds [Wed Apr 10 20:58:33 2019][2800032.606013] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Apr 10 20:58:34 2019][2800033.319687] LustreError: 117471:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955024, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b778ff9fbc0/0x53d0f21648f1fd0c lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f21648f1fd13 expref: -99 pid: 117471 timeout: 0 lvb_type: 0 [Wed Apr 10 20:58:34 2019][2800033.359960] LustreError: 117471:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 20:58:40 2019][2800039.439815] LustreError: 118262:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955030, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b743aa3c5c0/0x53d0f21648fc9c15 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f21648fc9c1c expref: -99 pid: 118262 timeout: 0 lvb_type: 0 [Wed Apr 10 20:58:52 2019][2800051.174064] LustreError: 117550:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955042, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b58a4f3b3c0/0x53d0f216490d5de8 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 18 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 117550 timeout: 0 lvb_type: 0 [Wed Apr 10 20:58:52 2019][2800051.746081] LustreError: 118171:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955042, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b523f29fbc0/0x53d0f216490e2369 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 10 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f216490e2370 expref: -99 pid: 118171 timeout: 0 lvb_type: 0 [Wed Apr 10 20:58:52 2019][2800051.786427] LustreError: 118171:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 20:59:18 2019][2800077.638630] LustreError: 118554:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955068, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b7913fcb840/0x53d0f21649318180 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 24 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118554 timeout: 0 lvb_type: 0 [Wed Apr 10 20:59:24 2019][2800083.756759] LustreError: 118755:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955074, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b5df8e26780/0x53d0f21649399e11 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 16 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f21649399e18 expref: -99 pid: 118755 timeout: 0 lvb_type: 0 [Wed Apr 10 20:59:32 2019][2800091.323920] Lustre: 118460:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 20:59:32 2019][2800091.323920] req@ffff8b72a6363f00 x1630475576920352/t0(0) o36->373dc3ac-323f-515c-a95e-226194580651@10.9.115.8@o2ib4:7/0 lens 560/2888 e 0 to 0 dl 1554955177 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 20:59:32 2019][2800091.352993] Lustre: 118460:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Apr 10 20:59:36 2019][2800095.567014] LustreError: 118212:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955086, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b8b13a7e300/0x53d0f216494d579c lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 26 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118212 timeout: 0 lvb_type: 0 [Wed Apr 10 20:59:48 2019][2800107.282261] LNet: Service thread pid 118272 was inactive for 200.49s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 20:59:48 2019][2800107.299454] LNet: Skipped 2 previous similar messages [Wed Apr 10 20:59:48 2019][2800107.304689] Pid: 118272, comm: mdt01_046 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 20:59:48 2019][2800107.314706] Call Trace: [Wed Apr 10 20:59:48 2019][2800107.317341] [] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Wed Apr 10 20:59:48 2019][2800107.324133] [] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] [Wed Apr 10 20:59:48 2019][2800107.330980] [] osp_remote_sync+0xd3/0x200 [osp] [Wed Apr 10 20:59:48 2019][2800107.337377] [] osp_xattr_get+0x70c/0x1100 [osp] [Wed Apr 10 20:59:48 2019][2800107.343778] [] lod_xattr_get+0xee/0x700 [lod] [Wed Apr 10 20:59:48 2019][2800107.350003] [] __mdd_links_read+0x118/0x3e0 [mdd] [Wed Apr 10 20:59:48 2019][2800107.356582] [] mdd_parent_fid+0xb8/0x3b0 [mdd] [Wed Apr 10 20:59:48 2019][2800107.362893] [] mdd_is_parent+0xd0/0x1a0 [mdd] [Wed Apr 10 20:59:48 2019][2800107.369124] [] mdd_is_subdir+0x204/0x240 [mdd] [Wed Apr 10 20:59:48 2019][2800107.375425] [] mdt_reint_rename+0x37f/0x28e0 [mdt] [Wed Apr 10 20:59:48 2019][2800107.382099] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 20:59:48 2019][2800107.388320] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 20:59:48 2019][2800107.395141] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 20:59:48 2019][2800107.401016] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 20:59:48 2019][2800107.408158] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 20:59:48 2019][2800107.416081] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 20:59:48 2019][2800107.422580] [] kthread+0xd1/0xe0 [Wed Apr 10 20:59:48 2019][2800107.427732] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 20:59:48 2019][2800107.434401] [] 0xffffffffffffffff [Wed Apr 10 20:59:48 2019][2800107.439613] LustreError: dumping log to /tmp/lustre-log.1554955188.118272 [Wed Apr 10 20:59:49 2019][2800108.855929] LNet: Service thread pid 118321 was inactive for 200.87s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 20:59:49 2019][2800108.873142] Pid: 118321, comm: mdt01_063 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 20:59:49 2019][2800108.883142] Call Trace: [Wed Apr 10 20:59:49 2019][2800108.885782] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 20:59:49 2019][2800108.892897] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 20:59:49 2019][2800108.900272] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Wed Apr 10 20:59:49 2019][2800108.906757] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 20:59:49 2019][2800108.913413] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 20:59:49 2019][2800108.919653] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 20:59:49 2019][2800108.926411] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 20:59:49 2019][2800108.932289] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 20:59:49 2019][2800108.939403] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 20:59:49 2019][2800108.947293] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 20:59:49 2019][2800108.953807] [] kthread+0xd1/0xe0 [Wed Apr 10 20:59:49 2019][2800108.958899] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 20:59:49 2019][2800108.965572] [] 0xffffffffffffffff [Wed Apr 10 20:59:53 2019][2800112.402370] LNet: Service thread pid 118393 was inactive for 200.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 20:59:53 2019][2800112.419568] Pid: 118393, comm: mdt01_100 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 20:59:53 2019][2800112.429566] Call Trace: [Wed Apr 10 20:59:53 2019][2800112.432206] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 20:59:53 2019][2800112.439313] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 20:59:53 2019][2800112.446594] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 20:59:53 2019][2800112.453455] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 20:59:53 2019][2800112.460200] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 20:59:53 2019][2800112.466595] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 20:59:53 2019][2800112.472906] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 20:59:53 2019][2800112.480343] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 20:59:53 2019][2800112.487259] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 20:59:53 2019][2800112.493680] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 20:59:53 2019][2800112.500337] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 20:59:53 2019][2800112.506560] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 20:59:53 2019][2800112.513303] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 20:59:53 2019][2800112.519176] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 20:59:53 2019][2800112.526292] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 20:59:53 2019][2800112.534197] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 20:59:53 2019][2800112.540698] [] kthread+0xd1/0xe0 [Wed Apr 10 20:59:53 2019][2800112.545799] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 20:59:53 2019][2800112.552451] [] 0xffffffffffffffff [Wed Apr 10 20:59:53 2019][2800112.557659] LustreError: dumping log to /tmp/lustre-log.1554955193.118393 [Wed Apr 10 20:59:55 2019][2800114.450416] LNet: Service thread pid 118253 was inactive for 200.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 20:59:55 2019][2800114.467967] Pid: 118253, comm: mdt00_040 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 20:59:55 2019][2800114.478285] Call Trace: [Wed Apr 10 20:59:55 2019][2800114.480939] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 20:59:55 2019][2800114.488332] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 20:59:55 2019][2800114.495615] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 20:59:55 2019][2800114.502488] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 20:59:55 2019][2800114.509233] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 20:59:55 2019][2800114.515699] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 20:59:55 2019][2800114.522016] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 20:59:55 2019][2800114.529468] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 20:59:55 2019][2800114.536385] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 20:59:55 2019][2800114.542798] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 20:59:55 2019][2800114.549454] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 20:59:55 2019][2800114.555678] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 20:59:55 2019][2800114.562454] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 20:59:55 2019][2800114.568389] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 20:59:55 2019][2800114.575542] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 20:59:55 2019][2800114.583437] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 20:59:55 2019][2800114.589976] [] kthread+0xd1/0xe0 [Wed Apr 10 20:59:55 2019][2800114.595069] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 20:59:55 2019][2800114.601743] [] 0xffffffffffffffff [Wed Apr 10 20:59:55 2019][2800114.606936] LustreError: dumping log to /tmp/lustre-log.1554955195.118253 [Wed Apr 10 20:59:57 2019][2800116.498461] Pid: 118364, comm: mdt02_044 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 20:59:57 2019][2800116.508469] Call Trace: [Wed Apr 10 20:59:57 2019][2800116.511110] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 20:59:57 2019][2800116.518276] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 20:59:57 2019][2800116.525663] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Wed Apr 10 20:59:57 2019][2800116.532156] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 20:59:57 2019][2800116.538828] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 20:59:57 2019][2800116.545055] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 20:59:57 2019][2800116.551811] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 20:59:57 2019][2800116.557691] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 20:59:57 2019][2800116.564838] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 20:59:57 2019][2800116.572738] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 20:59:57 2019][2800116.579260] [] kthread+0xd1/0xe0 [Wed Apr 10 20:59:57 2019][2800116.584357] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 20:59:57 2019][2800116.591023] [] 0xffffffffffffffff [Wed Apr 10 20:59:57 2019][2800116.596226] LustreError: dumping log to /tmp/lustre-log.1554955197.118364 [Wed Apr 10 21:00:13 2019][2800132.370799] LNet: Service thread pid 118541 was inactive for 200.46s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:00:13 2019][2800132.383926] LNet: Skipped 7 previous similar messages [Wed Apr 10 21:00:13 2019][2800132.389161] LustreError: dumping log to /tmp/lustre-log.1554955213.118541 [Wed Apr 10 21:00:20 2019][2800139.723955] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Apr 10 21:00:20 2019][2800139.739099] LustreError: 118470:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955130, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b7852702f40/0x53d0f21649980e19 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 18 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f21649980e3c expref: -99 pid: 118470 timeout: 0 lvb_type: 0 [Wed Apr 10 21:00:20 2019][2800139.779495] LustreError: 118470:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 21:00:20 2019][2800139.790663] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Apr 10 21:00:24 2019][2800143.635041] LNet: Service thread pid 117471 was inactive for 200.31s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:00:24 2019][2800143.648162] LustreError: dumping log to /tmp/lustre-log.1554955224.117471 [Wed Apr 10 21:00:30 2019][2800149.779178] LNet: Service thread pid 118262 was inactive for 200.33s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:00:30 2019][2800149.792302] LustreError: dumping log to /tmp/lustre-log.1554955230.118262 [Wed Apr 10 21:00:32 2019][2800151.828213] LNet: Service thread pid 118327 was inactive for 200.41s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:00:32 2019][2800151.841340] LustreError: dumping log to /tmp/lustre-log.1554955232.118327 [Wed Apr 10 21:00:39 2019][2800158.121361] LustreError: 117559:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955148, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b4d103245c0/0x53d0f21649bbb33a lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 32 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 117559 timeout: 0 lvb_type: 0 [Wed Apr 10 21:00:39 2019][2800158.160835] LustreError: 117559:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 21:00:42 2019][2800161.556423] LNet: Service thread pid 117550 was inactive for 200.37s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:00:42 2019][2800161.569548] LustreError: dumping log to /tmp/lustre-log.1554955242.117550 [Wed Apr 10 21:01:08 2019][2800187.667985] LNet: Service thread pid 118554 was inactive for 200.02s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:01:08 2019][2800187.681111] LNet: Skipped 1 previous similar message [Wed Apr 10 21:01:08 2019][2800187.686260] LustreError: dumping log to /tmp/lustre-log.1554955268.118554 [Wed Apr 10 21:01:14 2019][2800193.812122] LustreError: dumping log to /tmp/lustre-log.1554955274.118755 [Wed Apr 10 21:01:19 2019][2800198.932229] LustreError: dumping log to /tmp/lustre-log.1554955279.118184 [Wed Apr 10 21:01:26 2019][2800206.100380] LNet: Service thread pid 118212 was inactive for 200.53s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:01:26 2019][2800206.113527] LNet: Skipped 2 previous similar messages [Wed Apr 10 21:01:26 2019][2800206.118763] LustreError: dumping log to /tmp/lustre-log.1554955286.118212 [Wed Apr 10 21:01:35 2019][2800214.292555] LustreError: dumping log to /tmp/lustre-log.1554955295.118363 [Wed Apr 10 21:01:39 2019][2800218.747660] LustreError: 118585:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955209, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b79e1c56c00/0x53d0f2164a275ce8 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 34 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118585 timeout: 0 lvb_type: 0 [Wed Apr 10 21:01:39 2019][2800218.787158] LustreError: 118585:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Wed Apr 10 21:01:43 2019][2800222.360739] LustreError: 118574:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955213, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b87b1690000/0x53d0f2164a2c3458 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 22 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f2164a2c345f expref: -99 pid: 118574 timeout: 0 lvb_type: 0 [Wed Apr 10 21:01:43 2019][2800222.401101] LustreError: 118574:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 7 previous similar messages [Wed Apr 10 21:01:51 2019][2800230.342903] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Apr 10 21:01:51 2019][2800230.358118] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Apr 10 21:02:04 2019][2800243.713202] Lustre: 118297:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 21:02:04 2019][2800243.713202] req@ffff8b6351e69800 x1630376895289184/t0(0) o36->7661a650-b983-f969-4be5-b91e2d8da7f1@10.9.105.71@o2ib4:9/0 lens 600/2888 e 0 to 0 dl 1554955329 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:02:04 2019][2800243.742365] Lustre: 118297:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages [Wed Apr 10 21:02:11 2019][2800250.133333] LNet: Service thread pid 118470 was inactive for 200.40s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:02:11 2019][2800250.146452] LNet: Skipped 1 previous similar message [Wed Apr 10 21:02:11 2019][2800250.151601] LustreError: dumping log to /tmp/lustre-log.1554955331.118470 [Wed Apr 10 21:02:27 2019][2800266.518684] LustreError: dumping log to /tmp/lustre-log.1554955347.118431 [Wed Apr 10 21:02:28 2019][2800267.541706] LustreError: dumping log to /tmp/lustre-log.1554955348.118288 [Wed Apr 10 21:02:29 2019][2800268.565728] LustreError: dumping log to /tmp/lustre-log.1554955349.117559 [Wed Apr 10 21:02:37 2019][2800276.757903] LustreError: dumping log to /tmp/lustre-log.1554955357.117898 [Wed Apr 10 21:02:38 2019][2800277.270910] LustreError: dumping log to /tmp/lustre-log.1554955358.118247 [Wed Apr 10 21:02:39 2019][2800278.805947] LustreError: dumping log to /tmp/lustre-log.1554955359.117466 [Wed Apr 10 21:02:47 2019][2800286.998117] LustreError: dumping log to /tmp/lustre-log.1554955367.118331 [Wed Apr 10 21:02:59 2019][2800298.262365] LustreError: dumping log to /tmp/lustre-log.1554955379.118176 [Wed Apr 10 21:03:11 2019][2800310.601628] LNetError: 117113:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [Wed Apr 10 21:03:11 2019][2800310.611887] LNetError: 117113:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (55): c: 0, oc: 0, rc: 8 [Wed Apr 10 21:03:14 2019][2800313.934705] LustreError: 118312:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955304, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b639cfb60c0/0x53d0f2164ab2b77a lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 42 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118312 timeout: 0 lvb_type: 0 [Wed Apr 10 21:03:14 2019][2800313.974181] LustreError: 118312:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Wed Apr 10 21:03:29 2019][2800328.983023] LNet: Service thread pid 118585 was inactive for 200.23s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:03:29 2019][2800328.996383] LNet: Skipped 9 previous similar messages [Wed Apr 10 21:03:29 2019][2800329.001642] LustreError: dumping log to /tmp/lustre-log.1554955409.118585 [Wed Apr 10 21:03:30 2019][2800329.495036] LustreError: dumping log to /tmp/lustre-log.1554955410.118622 [Wed Apr 10 21:03:33 2019][2800333.079111] LustreError: dumping log to /tmp/lustre-log.1554955413.118574 [Wed Apr 10 21:03:39 2019][2800339.023235] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Apr 10 21:03:39 2019][2800339.038493] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Apr 10 21:03:41 2019][2800340.759279] LustreError: dumping log to /tmp/lustre-log.1554955421.118151 [Wed Apr 10 21:03:42 2019][2800341.271293] LustreError: dumping log to /tmp/lustre-log.1554955422.118245 [Wed Apr 10 21:04:05 2019][2800364.823804] LustreError: dumping log to /tmp/lustre-log.1554955445.118209 [Wed Apr 10 21:04:44 2019][2800403.612652] LustreError: 118403:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955394, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b6596a0da00/0x53d0f2164b4b6899 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 28 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f2164b4b68a0 expref: -99 pid: 118403 timeout: 0 lvb_type: 0 [Wed Apr 10 21:04:44 2019][2800403.652992] LustreError: 118403:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 6 previous similar messages [Wed Apr 10 21:04:59 2019][2800418.584957] LNet: Service thread pid 118437 was inactive for 200.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 21:04:59 2019][2800418.602156] LNet: Skipped 1 previous similar message [Wed Apr 10 21:04:59 2019][2800418.607306] Pid: 118437, comm: mdt01_109 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:04:59 2019][2800418.617308] Call Trace: [Wed Apr 10 21:04:59 2019][2800418.619954] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:04:59 2019][2800418.627058] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 21:04:59 2019][2800418.634357] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 21:04:59 2019][2800418.641203] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 21:04:59 2019][2800418.647970] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 21:04:59 2019][2800418.654388] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 21:04:59 2019][2800418.660697] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 21:04:59 2019][2800418.668148] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 21:04:59 2019][2800418.675080] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 21:04:59 2019][2800418.681484] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:04:59 2019][2800418.688143] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:04:59 2019][2800418.694366] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:04:59 2019][2800418.701123] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:04:59 2019][2800418.707003] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:04:59 2019][2800418.714172] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:04:59 2019][2800418.722104] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:04:59 2019][2800418.728617] [] kthread+0xd1/0xe0 [Wed Apr 10 21:04:59 2019][2800418.733706] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:04:59 2019][2800418.740353] [] 0xffffffffffffffff [Wed Apr 10 21:04:59 2019][2800418.745573] LustreError: dumping log to /tmp/lustre-log.1554955499.118437 [Wed Apr 10 21:04:59 2019][2800419.096976] Pid: 118370, comm: mdt01_092 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:04:59 2019][2800419.107035] Call Trace: [Wed Apr 10 21:04:59 2019][2800419.109678] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:04:59 2019][2800419.116789] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 21:05:00 2019][2800419.124067] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 21:05:00 2019][2800419.130914] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 21:05:00 2019][2800419.137649] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 21:05:00 2019][2800419.144068] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 21:05:00 2019][2800419.150398] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 21:05:00 2019][2800419.157835] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 21:05:00 2019][2800419.164771] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 21:05:00 2019][2800419.171183] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:05:00 2019][2800419.177854] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:05:00 2019][2800419.184080] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:05:00 2019][2800419.190824] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:05:00 2019][2800419.196713] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:05:00 2019][2800419.203934] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:05:00 2019][2800419.211867] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:05:00 2019][2800419.218382] [] kthread+0xd1/0xe0 [Wed Apr 10 21:05:00 2019][2800419.223469] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:05:00 2019][2800419.230143] [] 0xffffffffffffffff [Wed Apr 10 21:05:00 2019][2800419.235345] LustreError: dumping log to /tmp/lustre-log.1554955500.118370 [Wed Apr 10 21:05:00 2019][2800419.603993] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds [Wed Apr 10 21:05:00 2019][2800419.614340] LNet: 117113:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages [Wed Apr 10 21:05:05 2019][2800424.217107] Pid: 118312, comm: mdt02_038 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:05:05 2019][2800424.227111] Call Trace: [Wed Apr 10 21:05:05 2019][2800424.229753] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:05:05 2019][2800424.236858] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:05:05 2019][2800424.244215] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Wed Apr 10 21:05:05 2019][2800424.250699] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:05:05 2019][2800424.257346] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:05:05 2019][2800424.263560] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:05:05 2019][2800424.270323] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:05:05 2019][2800424.276186] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:05:05 2019][2800424.283319] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:05:05 2019][2800424.291206] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:05:05 2019][2800424.297706] [] kthread+0xd1/0xe0 [Wed Apr 10 21:05:05 2019][2800424.302785] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:05:05 2019][2800424.309434] [] 0xffffffffffffffff [Wed Apr 10 21:05:05 2019][2800424.314640] LustreError: dumping log to /tmp/lustre-log.1554955505.118312 [Wed Apr 10 21:05:20 2019][2800439.136476] Lustre: fir-MDT0000: haven't heard from client eec1cf9a-7f26-96fc-3f79-08df8cf1f446 (at 10.9.107.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b57c3a34000, cur 1554955520 expire 1554955370 last 1554955293 [Wed Apr 10 21:05:20 2019][2800439.158436] Lustre: Skipped 4 previous similar messages [Wed Apr 10 21:05:30 2019][2800449.305642] LNet: Service thread pid 118533 was inactive for 200.27s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 21:05:30 2019][2800449.322859] LNet: Skipped 2 previous similar messages [Wed Apr 10 21:05:30 2019][2800449.328092] Pid: 118533, comm: mdt02_092 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:05:30 2019][2800449.338105] Call Trace: [Wed Apr 10 21:05:30 2019][2800449.340743] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:05:30 2019][2800449.347864] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 21:05:30 2019][2800449.355161] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 21:05:30 2019][2800449.362008] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 21:05:30 2019][2800449.368767] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 21:05:30 2019][2800449.375168] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 21:05:30 2019][2800449.381475] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 21:05:30 2019][2800449.388927] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 21:05:30 2019][2800449.395861] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 21:05:30 2019][2800449.402258] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:05:30 2019][2800449.408944] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:05:30 2019][2800449.415172] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:05:30 2019][2800449.421923] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:05:30 2019][2800449.427800] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:05:30 2019][2800449.434947] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:05:30 2019][2800449.442846] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:05:30 2019][2800449.449360] [] kthread+0xd1/0xe0 [Wed Apr 10 21:05:30 2019][2800449.454448] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:05:30 2019][2800449.461098] [] 0xffffffffffffffff [Wed Apr 10 21:05:30 2019][2800449.466290] LustreError: dumping log to /tmp/lustre-log.1554955530.118533 [Wed Apr 10 21:05:38 2019][2800458.009832] Pid: 118735, comm: mdt03_094 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:05:38 2019][2800458.019830] Call Trace: [Wed Apr 10 21:05:38 2019][2800458.022490] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:05:38 2019][2800458.029611] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 21:05:38 2019][2800458.036921] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 21:05:38 2019][2800458.043771] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 21:05:38 2019][2800458.050521] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 21:05:38 2019][2800458.056920] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 21:05:38 2019][2800458.063245] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 21:05:38 2019][2800458.070683] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 21:05:38 2019][2800458.077631] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 21:05:38 2019][2800458.084029] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:05:38 2019][2800458.090687] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:05:38 2019][2800458.096941] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:05:38 2019][2800458.103692] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:05:38 2019][2800458.109570] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:05:39 2019][2800458.116710] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:05:39 2019][2800458.124623] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:05:39 2019][2800458.131140] [] kthread+0xd1/0xe0 [Wed Apr 10 21:05:39 2019][2800458.136232] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:05:39 2019][2800458.142891] [] 0xffffffffffffffff [Wed Apr 10 21:05:39 2019][2800458.148090] LustreError: dumping log to /tmp/lustre-log.1554955539.118735 [Wed Apr 10 21:05:47 2019][2800466.202011] LNet: Service thread pid 118660 was inactive for 200.31s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:05:47 2019][2800466.215132] LNet: Skipped 5 previous similar messages [Wed Apr 10 21:05:47 2019][2800466.220369] LustreError: dumping log to /tmp/lustre-log.1554955547.118660 [Wed Apr 10 21:06:31 2019][2800510.762990] Lustre: 118153:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 21:06:31 2019][2800510.762990] req@ffff8b52430b9b00 x1630383964599072/t0(0) o36->261ee164-624b-aba3-f37d-69ea5bb1f14f@10.9.112.17@o2ib4:6/0 lens 536/2888 e 0 to 0 dl 1554955596 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:06:31 2019][2800510.792155] Lustre: 118153:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages [Wed Apr 10 21:06:33 2019][2800512.283007] LustreError: dumping log to /tmp/lustre-log.1554955593.118232 [Wed Apr 10 21:06:35 2019][2800514.331054] LustreError: dumping log to /tmp/lustre-log.1554955595.118403 [Wed Apr 10 21:06:42 2019][2800521.657210] LustreError: 118281:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955512, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b6c0278a400/0x53d0f2164c1e4a59 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 49 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118281 timeout: 0 lvb_type: 0 [Wed Apr 10 21:06:42 2019][2800521.696702] LustreError: 118281:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Wed Apr 10 21:06:50 2019][2800529.691381] LustreError: dumping log to /tmp/lustre-log.1554955610.118338 [Wed Apr 10 21:07:14 2019][2800553.486123] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Wed Apr 10 21:07:14 2019][2800553.493530] Lustre: Skipped 1685 previous similar messages [Wed Apr 10 21:07:30 2019][2800570.022614] Lustre: fir-MDT0002: Client b7d5a963-1bd8-0aaa-de83-eb1f005c0685 (at 10.9.102.2@o2ib4) reconnecting [Wed Apr 10 21:07:30 2019][2800570.032886] Lustre: Skipped 1687 previous similar messages [Wed Apr 10 21:07:31 2019][2800570.607267] LNetError: 117113:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [Wed Apr 10 21:07:31 2019][2800570.617529] LNetError: 117113:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (3): c: 0, oc: 0, rc: 8 [Wed Apr 10 21:07:53 2019][2800593.111831] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.9.105.29@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [Wed Apr 10 21:07:54 2019][2800593.129377] LustreError: Skipped 19319 previous similar messages [Wed Apr 10 21:08:09 2019][2800608.997092] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Wed Apr 10 21:08:09 2019][2800609.012314] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Wed Apr 10 21:08:32 2019][2800632.093599] LustreError: dumping log to /tmp/lustre-log.1554955712.118281 [Wed Apr 10 21:09:27 2019][2800686.366751] LustreError: dumping log to /tmp/lustre-log.1554955767.118536 [Wed Apr 10 21:09:39 2019][2800699.055277] LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [Wed Apr 10 21:09:40 2019][2800699.055277] [Wed Apr 10 21:09:40 2019][2800699.061114] LDISKFS-fs warning (device dm-4): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [Wed Apr 10 21:09:40 2019][2800699.061114] [Wed Apr 10 21:09:59 2019][2800719.135455] LNet: Service thread pid 118240 was inactive for 200.13s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 21:10:00 2019][2800719.152652] LNet: Skipped 1 previous similar message [Wed Apr 10 21:10:00 2019][2800719.157801] Pid: 118240, comm: mdt01_035 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:10:00 2019][2800719.167797] Call Trace: [Wed Apr 10 21:10:00 2019][2800719.170446] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:10:00 2019][2800719.177584] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 21:10:00 2019][2800719.184869] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 21:10:00 2019][2800719.191758] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 21:10:00 2019][2800719.198500] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 21:10:00 2019][2800719.204910] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 21:10:00 2019][2800719.211224] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 21:10:00 2019][2800719.218675] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 21:10:00 2019][2800719.225595] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 21:10:00 2019][2800719.232003] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:10:00 2019][2800719.238663] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:10:00 2019][2800719.244885] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:10:00 2019][2800719.251629] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:10:00 2019][2800719.257533] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:10:00 2019][2800719.264676] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:10:00 2019][2800719.272584] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:10:00 2019][2800719.279107] [] kthread+0xd1/0xe0 [Wed Apr 10 21:10:00 2019][2800719.284199] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:10:00 2019][2800719.290859] [] 0xffffffffffffffff [Wed Apr 10 21:10:00 2019][2800719.296056] LustreError: dumping log to /tmp/lustre-log.1554955800.118240 [Wed Apr 10 21:10:01 2019][2800720.159472] Pid: 118394, comm: mdt00_068 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:10:01 2019][2800720.169468] Call Trace: [Wed Apr 10 21:10:01 2019][2800720.172110] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:10:01 2019][2800720.179265] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 21:10:01 2019][2800720.186590] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 21:10:01 2019][2800720.193441] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 21:10:01 2019][2800720.200205] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 21:10:01 2019][2800720.206605] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 21:10:01 2019][2800720.212930] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 21:10:01 2019][2800720.220371] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 21:10:01 2019][2800720.227310] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 21:10:01 2019][2800720.233708] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:10:01 2019][2800720.240378] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:10:01 2019][2800720.246618] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:10:01 2019][2800720.253363] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:10:01 2019][2800720.259239] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:10:01 2019][2800720.266380] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:10:01 2019][2800720.274278] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:10:01 2019][2800720.280793] [] kthread+0xd1/0xe0 [Wed Apr 10 21:10:01 2019][2800720.285881] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:10:01 2019][2800720.292544] [] 0xffffffffffffffff [Wed Apr 10 21:10:01 2019][2800720.297750] LustreError: dumping log to /tmp/lustre-log.1554955801.118394 [Wed Apr 10 21:10:21 2019][2800741.062203] LDISKFS-fs (dm-4): file extents enabled, maximum tree depth=5 [Wed Apr 10 21:10:21 2019][2800741.083169] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [Wed Apr 10 21:10:28 2019][2800747.149314] LDISKFS-fs (dm-2): recovery complete [Wed Apr 10 21:10:28 2019][2800747.154576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [Wed Apr 10 21:10:28 2019][2800747.706491] Lustre: fir-MDT0003: Not available for connect from 10.9.104.57@o2ib4 (not set up) [Wed Apr 10 21:10:28 2019][2800747.715323] Lustre: Skipped 65 previous similar messages [Wed Apr 10 21:10:28 2019][2800748.041986] Lustre: fir-MDT0000: Received LWP connection from 0@lo, removing former export from 10.0.10.52@o2ib7 [Wed Apr 10 21:10:28 2019][2800748.047292] Lustre: fir-MDT0003: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [Wed Apr 10 21:10:28 2019][2800748.050545] Lustre: fir-MDD0003: changelog on [Wed Apr 10 21:10:28 2019][2800748.050548] Lustre: Skipped 1 previous similar message [Wed Apr 10 21:10:28 2019][2800748.055258] Lustre: fir-MDT0003: in recovery but waiting for the first client to connect [Wed Apr 10 21:10:28 2019][2800748.055260] Lustre: Skipped 1 previous similar message [Wed Apr 10 21:10:29 2019][2800748.681357] Lustre: fir-MDT0003: Will be in recovery for at least 2:30, or until 1332 clients reconnect [Wed Apr 10 21:10:31 2019][2800750.880147] Pid: 118265, comm: mdt01_043 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:10:31 2019][2800750.890145] Call Trace: [Wed Apr 10 21:10:31 2019][2800750.892794] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:10:31 2019][2800750.899939] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Wed Apr 10 21:10:31 2019][2800750.907226] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Wed Apr 10 21:10:31 2019][2800750.914098] [] osp_md_object_lock+0x162/0x2d0 [osp] [Wed Apr 10 21:10:31 2019][2800750.920842] [] lod_object_lock+0xf3/0x7b0 [lod] [Wed Apr 10 21:10:31 2019][2800750.927237] [] mdd_object_lock+0x3e/0xe0 [mdd] [Wed Apr 10 21:10:31 2019][2800750.933547] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Wed Apr 10 21:10:31 2019][2800750.940992] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Wed Apr 10 21:10:31 2019][2800750.947967] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Wed Apr 10 21:10:31 2019][2800750.954365] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:10:31 2019][2800750.961047] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:10:31 2019][2800750.967270] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:10:31 2019][2800750.974030] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:10:31 2019][2800750.979906] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:10:31 2019][2800750.987032] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:10:31 2019][2800750.994927] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:10:31 2019][2800751.001684] [] kthread+0xd1/0xe0 [Wed Apr 10 21:10:31 2019][2800751.006773] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:10:31 2019][2800751.013436] [] 0xffffffffffffffff [Wed Apr 10 21:10:31 2019][2800751.018632] LustreError: dumping log to /tmp/lustre-log.1554955831.118265 [Wed Apr 10 21:10:33 2019][2800752.928178] LNet: Service thread pid 118399 was inactive for 200.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 21:10:33 2019][2800752.945418] LNet: Skipped 2 previous similar messages [Wed Apr 10 21:10:33 2019][2800752.950672] Pid: 118399, comm: mdt00_070 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:10:33 2019][2800752.960687] Call Trace: [Wed Apr 10 21:10:33 2019][2800752.963334] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:10:33 2019][2800752.970461] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:10:33 2019][2800752.977867] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Wed Apr 10 21:10:33 2019][2800752.984354] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:10:33 2019][2800752.991010] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:10:33 2019][2800752.997232] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:10:33 2019][2800753.003997] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:10:33 2019][2800753.009879] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:10:33 2019][2800753.017031] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:10:33 2019][2800753.024964] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:10:33 2019][2800753.031476] [] kthread+0xd1/0xe0 [Wed Apr 10 21:10:33 2019][2800753.036587] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:10:33 2019][2800753.043237] [] 0xffffffffffffffff [Wed Apr 10 21:10:33 2019][2800753.048464] LustreError: dumping log to /tmp/lustre-log.1554955833.118399 [Wed Apr 10 21:10:34 2019][2800753.664205] LDISKFS-fs (dm-4): recovery complete [Wed Apr 10 21:10:34 2019][2800753.673155] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [Wed Apr 10 21:10:36 2019][2800755.327396] Lustre: fir-MDT0000: Received LWP connection from 0@lo, removing former export from 10.0.10.52@o2ib7 [Wed Apr 10 21:10:36 2019][2800755.329503] Lustre: fir-MDT0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [Wed Apr 10 21:10:36 2019][2800755.333161] Lustre: fir-MDD0001: changelog on [Wed Apr 10 21:10:36 2019][2800755.338988] Lustre: fir-MDT0001: in recovery but waiting for the first client to connect [Wed Apr 10 21:10:36 2019][2800755.342221] Lustre: fir-MDT0001: Will be in recovery for at least 2:30, or until 1317 clients reconnect [Wed Apr 10 21:10:36 2019][2800755.468383] Lustre: fir-MDT0001: Denying connection for new client ec2bf5c6-c1ab-4f7e-c1ec-9de91b1694ce(at 10.9.101.54@o2ib4), waiting for 1317 known clients (4 recovered, 3 in progress, and 0 evicted) already passed deadline 2:30 [Wed Apr 10 21:10:37 2019][2800756.182038] Lustre: fir-MDT0001: Denying connection for new client 01faf8f4-b5fb-0e89-bb2f-a1b48c80ac0a(at 10.9.101.53@o2ib4), waiting for 1317 known clients (20 recovered, 9 in progress, and 0 evicted) already passed deadline 2:31 [Wed Apr 10 21:10:37 2019][2800756.202696] Lustre: Skipped 1 previous similar message [Wed Apr 10 21:10:38 2019][2800757.292103] Lustre: fir-MDT0001: Denying connection for new client 3915e1c9-c15f-afc0-874b-82dc91089b9a(at 10.9.112.10@o2ib4), waiting for 1317 known clients (41 recovered, 29 in progress, and 0 evicted) already passed deadline 2:32 [Wed Apr 10 21:10:38 2019][2800757.312876] Lustre: Skipped 2 previous similar messages [Wed Apr 10 21:10:40 2019][2800759.322706] Lustre: fir-MDT0001: Denying connection for new client e828a575-26d3-3703-3e8e-f85195978091(at 10.9.101.56@o2ib4), waiting for 1317 known clients (63 recovered, 40 in progress, and 0 evicted) already passed deadline 2:34 [Wed Apr 10 21:10:40 2019][2800759.343453] Lustre: Skipped 1 previous similar message [Wed Apr 10 21:10:45 2019][2800764.761881] Lustre: fir-MDT0001: Denying connection for new client bd276746-c827-b471-321d-30c1af34c941(at 10.9.114.15@o2ib4), waiting for 1317 known clients (99 recovered, 81 in progress, and 0 evicted) already passed deadline 2:39 [Wed Apr 10 21:10:45 2019][2800764.782637] Lustre: Skipped 1 previous similar message [Wed Apr 10 21:10:54 2019][2800773.960783] Lustre: DEBUG MARKER: Wed Apr 10 21:10:54 2019 [Wed Apr 10 21:10:54 2019][2800773.960783] [Wed Apr 10 21:10:57 2019][2800776.388603] Lustre: fir-MDT0001: Denying connection for new client 01faf8f4-b5fb-0e89-bb2f-a1b48c80ac0a(at 10.9.101.53@o2ib4), waiting for 1317 known clients (480 recovered, 100 in progress, and 0 evicted) already passed deadline 2:51 [Wed Apr 10 21:10:57 2019][2800776.409534] Lustre: Skipped 2 previous similar messages [Wed Apr 10 21:11:14 2019][2800793.699169] Lustre: fir-MDT0001: Denying connection for new client 29341325-8995-2420-e03f-e5ee976b3d1a(at 10.9.101.50@o2ib4), waiting for 1317 known clients (1135 recovered, 109 in progress, and 0 evicted) already passed deadline 3:08 [Wed Apr 10 21:11:14 2019][2800793.720178] Lustre: Skipped 8 previous similar messages [Wed Apr 10 21:11:27 2019][2800806.862340] LustreError: 118503:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554955797, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b4de950a1c0/0x53d0f2164e0e2d66 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 56 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118503 timeout: 0 lvb_type: 0 [Wed Apr 10 21:11:27 2019][2800806.901843] LustreError: 118503:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 5 previous similar messages [Wed Apr 10 21:11:31 2019][2800810.273406] Pid: 118366, comm: mdt01_090 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:11:31 2019][2800810.283411] Call Trace: [Wed Apr 10 21:11:31 2019][2800810.286057] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:11:31 2019][2800810.293174] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:11:31 2019][2800810.300540] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Wed Apr 10 21:11:31 2019][2800810.307023] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Wed Apr 10 21:11:31 2019][2800810.313679] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:11:31 2019][2800810.319918] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:11:31 2019][2800810.326669] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:11:31 2019][2800810.332547] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:11:31 2019][2800810.339680] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:11:31 2019][2800810.347604] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:11:31 2019][2800810.354115] [] kthread+0xd1/0xe0 [Wed Apr 10 21:11:31 2019][2800810.359207] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:11:31 2019][2800810.365870] [] 0xffffffffffffffff [Wed Apr 10 21:11:31 2019][2800810.371065] LustreError: dumping log to /tmp/lustre-log.1554955891.118366 [Wed Apr 10 21:11:37 2019][2800816.417538] LNet: Service thread pid 117532 was inactive for 200.71s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Wed Apr 10 21:11:37 2019][2800816.430663] LNet: Skipped 8 previous similar messages [Wed Apr 10 21:11:37 2019][2800816.435897] LustreError: dumping log to /tmp/lustre-log.1554955897.117532 [Wed Apr 10 21:11:46 2019][2800825.959210] Lustre: fir-MDT0001: Denying connection for new client 43450a53-c691-3edf-521c-21ca072c80c2(at 10.9.113.13@o2ib4), waiting for 1317 known clients (1191 recovered, 118 in progress, and 0 evicted) already passed deadline 3:41 [Wed Apr 10 21:11:46 2019][2800825.980218] Lustre: Skipped 20 previous similar messages [Wed Apr 10 21:12:39 2019][2800878.882906] LustreError: dumping log to /tmp/lustre-log.1554955959.118406 [Wed Apr 10 21:12:51 2019][2800890.359503] Lustre: fir-MDT0001: Denying connection for new client bee3e714-f82b-2aef-331b-b7b14bbce2b9(at 10.9.101.49@o2ib4), waiting for 1317 known clients (1191 recovered, 118 in progress, and 0 evicted) already passed deadline 4:45 [Wed Apr 10 21:12:51 2019][2800890.380519] Lustre: Skipped 40 previous similar messages [Wed Apr 10 21:12:59 2019][2800898.851396] Lustre: fir-MDT0003: recovery is timed out, evict stale exports [Wed Apr 10 21:12:59 2019][2800898.858891] Lustre: fir-MDT0003: disconnecting 7 stale clients [Wed Apr 10 21:12:59 2019][2800898.914052] Lustre: fir-MDT0003: Recovery already passed deadline 4:00, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. [Wed Apr 10 21:12:59 2019][2800898.930555] Lustre: Skipped 7 previous similar messages [Wed Apr 10 21:12:59 2019][2800898.982090] Lustre: fir-MDT0003: Recovery over after 2:30, of 1332 clients 1325 recovered and 7 were evicted. [Wed Apr 10 21:12:59 2019][2800898.992795] Lustre: Skipped 1 previous similar message [Wed Apr 10 21:13:06 2019][2800905.507528] Lustre: fir-MDT0001: recovery is timed out, evict stale exports [Wed Apr 10 21:13:06 2019][2800905.515042] Lustre: fir-MDT0001: disconnecting 8 stale clients [Wed Apr 10 21:13:09 2019][2800908.450444] LNet: Service thread pid 118272 completed after 1001.64s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Wed Apr 10 21:13:09 2019][2800908.467191] LNet: Skipped 8 previous similar messages [Wed Apr 10 21:13:09 2019][2800908.481340] Lustre: 118212:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (902:1s); client may timeout. req@ffff8b655fa26600 x1630452519977632/t155122092688(0) o36->392d8331-072b-04ee-380d-4c214c504d41@10.9.101.40@o2ib4:26/0 lens 776/424 e 1 to 0 dl 1554955988 ref 1 fl Complete:/0/0 rc 0/0 [Wed Apr 10 21:13:09 2019][2800908.510149] Lustre: fir-MDT0001: Recovery over after 2:33, of 1317 clients 1309 recovered and 8 were evicted. [Wed Apr 10 21:16:34 2019][2801113.384035] LNet: Service thread pid 118242 was inactive for 200.76s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 21:16:34 2019][2801113.401232] LNet: Skipped 1 previous similar message [Wed Apr 10 21:16:34 2019][2801113.406387] Pid: 118242, comm: mdt01_036 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:16:34 2019][2801113.416387] Call Trace: [Wed Apr 10 21:16:34 2019][2801113.419036] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:16:34 2019][2801113.426179] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:16:34 2019][2801113.433600] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 21:16:34 2019][2801113.440617] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 21:16:34 2019][2801113.447821] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 21:16:34 2019][2801113.454493] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 21:16:34 2019][2801113.461677] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 21:16:34 2019][2801113.468440] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 21:16:34 2019][2801113.475121] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 21:16:34 2019][2801113.482061] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 21:16:34 2019][2801113.489341] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 21:16:34 2019][2801113.495686] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:16:34 2019][2801113.502838] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:16:34 2019][2801113.510759] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:16:34 2019][2801113.517273] [] kthread+0xd1/0xe0 [Wed Apr 10 21:16:34 2019][2801113.522363] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:16:34 2019][2801113.529012] [] 0xffffffffffffffff [Wed Apr 10 21:16:34 2019][2801113.534211] LustreError: dumping log to /tmp/lustre-log.1554956194.118242 [Wed Apr 10 21:16:35 2019][2801114.640989] Pid: 118238, comm: mdt02_024 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:16:35 2019][2801114.650992] Call Trace: [Wed Apr 10 21:16:35 2019][2801114.653636] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.660766] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.668182] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 21:16:35 2019][2801114.675117] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 21:16:35 2019][2801114.681781] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:16:35 2019][2801114.688002] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:16:35 2019][2801114.694748] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:16:35 2019][2801114.700621] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.707747] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.715641] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.722140] [] kthread+0xd1/0xe0 [Wed Apr 10 21:16:35 2019][2801114.727222] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:16:35 2019][2801114.733868] [] 0xffffffffffffffff [Wed Apr 10 21:16:35 2019][2801114.739063] Pid: 118297, comm: mdt01_055 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:16:35 2019][2801114.749068] Call Trace: [Wed Apr 10 21:16:35 2019][2801114.751703] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.758820] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.766187] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 21:16:35 2019][2801114.773190] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 21:16:35 2019][2801114.780366] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 21:16:35 2019][2801114.787038] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 21:16:35 2019][2801114.794215] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 21:16:35 2019][2801114.800958] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 21:16:35 2019][2801114.807684] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.814617] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.821906] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.828249] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.835365] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.843262] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:16:35 2019][2801114.849770] [] kthread+0xd1/0xe0 [Wed Apr 10 21:16:35 2019][2801114.854858] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:16:35 2019][2801114.861506] [] 0xffffffffffffffff [Wed Apr 10 21:16:37 2019][2801116.968114] Pid: 118417, comm: mdt02_054 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:16:37 2019][2801116.978116] Call Trace: [Wed Apr 10 21:16:37 2019][2801116.980757] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801116.987881] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801116.995247] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 21:16:37 2019][2801117.002259] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 21:16:37 2019][2801117.009443] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 21:16:37 2019][2801117.016098] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 21:16:37 2019][2801117.023309] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 21:16:37 2019][2801117.030051] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 21:16:37 2019][2801117.036709] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801117.043651] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801117.050931] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801117.057267] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801117.064381] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801117.072276] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801117.078776] [] kthread+0xd1/0xe0 [Wed Apr 10 21:16:37 2019][2801117.083863] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:16:37 2019][2801117.090511] [] 0xffffffffffffffff [Wed Apr 10 21:16:37 2019][2801117.095708] LustreError: dumping log to /tmp/lustre-log.1554956197.118417 [Wed Apr 10 21:16:37 2019][2801117.121005] Pid: 118304, comm: mdt02_036 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:16:37 2019][2801117.131010] Call Trace: [Wed Apr 10 21:16:37 2019][2801117.133651] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:16:37 2019][2801117.140762] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:16:38 2019][2801117.148137] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 21:16:38 2019][2801117.155063] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 21:16:38 2019][2801117.161718] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:16:38 2019][2801117.167941] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:16:38 2019][2801117.174684] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:16:38 2019][2801117.180560] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:16:38 2019][2801117.187693] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:16:38 2019][2801117.195611] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:16:38 2019][2801117.202123] [] kthread+0xd1/0xe0 [Wed Apr 10 21:16:38 2019][2801117.207212] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:16:38 2019][2801117.213857] [] 0xffffffffffffffff [Wed Apr 10 21:16:47 2019][2801126.175884] Lustre: MGS: haven't heard from client 9e8d1cd1-b107-c792-e711-bd6c10a61564 (at 10.8.24.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5c87729400, cur 1554956207 expire 1554956057 last 1554955980 [Wed Apr 10 21:16:47 2019][2801126.197069] Lustre: Skipped 2 previous similar messages [Wed Apr 10 21:17:15 2019][2801154.972306] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Wed Apr 10 21:17:15 2019][2801154.982663] Lustre: Skipped 4472 previous similar messages [Wed Apr 10 21:17:31 2019][2801170.160523] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Wed Apr 10 21:17:31 2019][2801170.170876] Lustre: Skipped 1710 previous similar messages [Wed Apr 10 21:18:24 2019][2801223.444034] LustreError: 118811:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001083f:0xd145:0x0] [Wed Apr 10 21:20:27 2019][2801346.236668] LustreError: 119201:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5deec55400 x1628639176378448/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:27/0 lens 448/440 e 0 to 0 dl 1554956457 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:21:57 2019][2801436.184843] Lustre: 118419:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Wed Apr 10 21:21:57 2019][2801436.184843] req@ffff8b5898ab8000 x1628581971329328/t146178368963(0) o36->a634883e-678f-acdd-a857-3c8ad28c3204@10.9.106.34@o2ib4:2/0 lens 544/2888 e 0 to 0 dl 1554956522 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:21:57 2019][2801436.214974] Lustre: 118419:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 16 previous similar messages [Wed Apr 10 21:23:02 2019][2801501.943276] LustreError: 118575:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554956492, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b778e270900/0x53d0f2165c5eea36 lrc: 3/1,0 mode: --/PR res: [0x2c00104c8:0x7334:0x0].0x0 bits 0x13/0x8 rrc: 17 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118575 timeout: 0 lvb_type: 0 [Wed Apr 10 21:23:02 2019][2801501.983098] LustreError: 118575:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 7 previous similar messages [Wed Apr 10 21:24:52 2019][2801612.082624] LNet: Service thread pid 118409 was inactive for 200.13s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 21:24:52 2019][2801612.099835] LNet: Skipped 4 previous similar messages [Wed Apr 10 21:24:52 2019][2801612.105068] Pid: 118409, comm: mdt00_072 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:24:52 2019][2801612.115067] Call Trace: [Wed Apr 10 21:24:52 2019][2801612.117710] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:24:52 2019][2801612.124870] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:24:52 2019][2801612.132250] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 21:24:52 2019][2801612.139176] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 21:24:52 2019][2801612.145831] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:24:53 2019][2801612.152074] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:24:53 2019][2801612.158820] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:24:53 2019][2801612.164698] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:24:53 2019][2801612.171830] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:24:53 2019][2801612.179727] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:24:53 2019][2801612.186236] [] kthread+0xd1/0xe0 [Wed Apr 10 21:24:53 2019][2801612.191321] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:24:53 2019][2801612.197971] [] 0xffffffffffffffff [Wed Apr 10 21:24:53 2019][2801612.203174] LustreError: dumping log to /tmp/lustre-log.1554956693.118409 [Wed Apr 10 21:24:54 2019][2801613.438674] Pid: 118575, comm: mdt02_099 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:24:54 2019][2801613.448694] Call Trace: [Wed Apr 10 21:24:54 2019][2801613.451338] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:24:54 2019][2801613.458481] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:24:54 2019][2801613.465866] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 21:24:54 2019][2801613.472868] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 21:24:54 2019][2801613.480044] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 21:24:54 2019][2801613.486698] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 21:24:54 2019][2801613.493876] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 21:24:54 2019][2801613.500618] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 21:24:54 2019][2801613.507273] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 21:24:54 2019][2801613.514216] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 21:24:54 2019][2801613.521524] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 21:24:54 2019][2801613.527894] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:24:54 2019][2801613.535033] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:24:54 2019][2801613.542975] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:24:54 2019][2801613.549508] [] kthread+0xd1/0xe0 [Wed Apr 10 21:24:54 2019][2801613.554596] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:24:54 2019][2801613.561260] [] 0xffffffffffffffff [Wed Apr 10 21:27:16 2019][2801755.837097] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.23@o2ib6) [Wed Apr 10 21:27:16 2019][2801755.844418] Lustre: Skipped 1443 previous similar messages [Wed Apr 10 21:27:31 2019][2801770.949078] Lustre: fir-MDT0002: Client b477f0c2-cfda-70b4-919f-fe195c63921e (at 10.8.7.22@o2ib6) reconnecting [Wed Apr 10 21:27:31 2019][2801770.959254] Lustre: Skipped 1443 previous similar messages [Wed Apr 10 21:27:48 2019][2801787.305900] LustreError: 118085:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6031fd1e00 x1628639178159184/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:24/0 lens 448/440 e 0 to 0 dl 1554956874 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:30:17 2019][2801936.697477] LNet: Service thread pid 117470 was inactive for 200.24s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 21:30:17 2019][2801936.714678] LNet: Skipped 1 previous similar message [Wed Apr 10 21:30:17 2019][2801936.719823] Pid: 117470, comm: mdt02_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:30:17 2019][2801936.729822] Call Trace: [Wed Apr 10 21:30:17 2019][2801936.732483] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:30:17 2019][2801936.739619] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:30:17 2019][2801936.746996] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 21:30:17 2019][2801936.753937] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 21:30:17 2019][2801936.760596] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 21:30:17 2019][2801936.766818] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 21:30:17 2019][2801936.773563] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 21:30:17 2019][2801936.779445] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:30:17 2019][2801936.786570] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:30:17 2019][2801936.794461] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:30:17 2019][2801936.800975] [] kthread+0xd1/0xe0 [Wed Apr 10 21:30:17 2019][2801936.806063] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:30:17 2019][2801936.812710] [] 0xffffffffffffffff [Wed Apr 10 21:30:17 2019][2801936.817911] LustreError: dumping log to /tmp/lustre-log.1554957017.117470 [Wed Apr 10 21:30:18 2019][2801937.802456] Pid: 117911, comm: mdt02_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 21:30:18 2019][2801937.812460] Call Trace: [Wed Apr 10 21:30:18 2019][2801937.815103] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 21:30:18 2019][2801937.822234] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 21:30:18 2019][2801937.829605] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 21:30:18 2019][2801937.836627] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 21:30:18 2019][2801937.843810] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 21:30:18 2019][2801937.850475] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 21:30:18 2019][2801937.857659] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 21:30:18 2019][2801937.864420] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 21:30:18 2019][2801937.871094] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 21:30:18 2019][2801937.878044] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 21:30:18 2019][2801937.885333] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 21:30:18 2019][2801937.891703] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 21:30:18 2019][2801937.898836] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 21:30:18 2019][2801937.906741] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 21:30:18 2019][2801937.913257] [] kthread+0xd1/0xe0 [Wed Apr 10 21:30:18 2019][2801937.918352] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 21:30:18 2019][2801937.925000] [] 0xffffffffffffffff [Wed Apr 10 21:33:36 2019][2802135.173798] Lustre: fir-MDT0002: haven't heard from client 93087d26-40f2-f1d3-d998-4d73b4246b1f (at 10.9.108.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b79381b5c00, cur 1554957216 expire 1554957066 last 1554956989 [Wed Apr 10 21:33:36 2019][2802135.195765] Lustre: Skipped 9 previous similar messages [Wed Apr 10 21:34:04 2019][2802163.185519] LustreError: 119055:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b6c813d1850 x1629295271056720/t0(0) o4->56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5@10.8.8.8@o2ib6:10/0 lens 488/448 e 0 to 0 dl 1554957250 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:34:04 2019][2802163.210074] Lustre: fir-MDT0002: Bulk IO write error with 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6), client will retry: rc = -107 [Wed Apr 10 21:36:35 2019][2802314.347938] Lustre: 118559:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Apr 10 21:36:35 2019][2802314.881529] Lustre: 118270:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Wed Apr 10 21:36:35 2019][2802314.893444] Lustre: 118270:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 67 previous similar messages [Wed Apr 10 21:37:16 2019][2802356.052804] Lustre: fir-MDT0002: Connection restored to f7850cdc-3473-bc74-2a28-406fa0617404 (at 10.9.108.67@o2ib4) [Wed Apr 10 21:37:16 2019][2802356.063483] Lustre: Skipped 1510 previous similar messages [Wed Apr 10 21:37:34 2019][2802373.190023] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Wed Apr 10 21:37:34 2019][2802373.200119] Lustre: Skipped 1502 previous similar messages [Wed Apr 10 21:40:18 2019][2802537.293490] LustreError: 119183:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6107f7ef00 x1629298924624256/t0(0) o37->40c21896-2b28-d1d6-5611-05f09dc83863@10.9.108.24@o2ib4:24/0 lens 448/440 e 0 to 0 dl 1554957624 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:42:09 2019][2802648.421865] LustreError: 118082:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8bbd3ab000 x1628639180313408/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:15/0 lens 448/440 e 0 to 0 dl 1554957735 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:47:17 2019][2802956.715690] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Wed Apr 10 21:47:17 2019][2802956.723006] Lustre: Skipped 1510 previous similar messages [Wed Apr 10 21:47:34 2019][2802974.034161] Lustre: fir-MDT0002: Client 83b8ff7e-e264-e3fb-e7eb-42bd933886ea (at 10.8.1.23@o2ib6) reconnecting [Wed Apr 10 21:47:34 2019][2802974.044344] Lustre: Skipped 1506 previous similar messages [Wed Apr 10 21:51:57 2019][2803236.918074] LustreError: 117475:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b565c04ad00 x1628639181471776/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:3/0 lens 448/440 e 0 to 0 dl 1554958323 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:52:18 2019][2803257.530768] LustreError: 118026:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b681c6e6c50 x1628639181517760/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:24/0 lens 448/440 e 0 to 0 dl 1554958344 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 21:56:49 2019][2803528.395198] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Wed Apr 10 21:57:17 2019][2803556.854851] Lustre: fir-MDT0002: Connection restored to a98aeccc-0553-403d-5376-03640bbd1236 (at 10.9.107.61@o2ib4) [Wed Apr 10 21:57:17 2019][2803556.865467] Lustre: Skipped 1510 previous similar messages [Wed Apr 10 21:57:35 2019][2803574.433867] Lustre: fir-MDT0002: Client 99b2b05a-80e4-edd2-5271-b136b8e4528d (at 10.9.108.67@o2ib4) reconnecting [Wed Apr 10 21:57:35 2019][2803574.444212] Lustre: Skipped 1504 previous similar messages [Wed Apr 10 22:07:19 2019][2804158.453869] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Wed Apr 10 22:07:19 2019][2804158.464226] Lustre: Skipped 1504 previous similar messages [Wed Apr 10 22:07:35 2019][2804175.158671] Lustre: fir-MDT0002: Client 35ddd205-c174-6b95-20ef-c38a3193733f (at 10.9.108.35@o2ib4) reconnecting [Wed Apr 10 22:07:35 2019][2804175.169028] Lustre: Skipped 1502 previous similar messages [Wed Apr 10 22:07:37 2019][2804176.216138] Lustre: fir-MDT0002: haven't heard from client 61c46f26-ff19-795c-d693-58f9c03cf0a6 (at 10.9.113.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b791c37a800, cur 1554959257 expire 1554959107 last 1554959030 [Wed Apr 10 22:07:37 2019][2804176.238127] Lustre: Skipped 4 previous similar messages [Wed Apr 10 22:12:03 2019][2804443.120910] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Wed Apr 10 22:17:09 2019][2804748.766916] LustreError: 118027:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b73f02f8c50 x1628639185565984/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:15/0 lens 448/440 e 0 to 0 dl 1554959835 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 22:17:20 2019][2804759.300222] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.23@o2ib6) [Wed Apr 10 22:17:20 2019][2804759.307537] Lustre: Skipped 1504 previous similar messages [Wed Apr 10 22:17:36 2019][2804775.217760] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Wed Apr 10 22:17:36 2019][2804775.227937] Lustre: Skipped 1507 previous similar messages [Wed Apr 10 22:18:12 2019][2804811.769767] LustreError: 119252:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b4e55fa4200 x1628639185681856/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:18/0 lens 448/440 e 0 to 0 dl 1554959898 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 22:24:22 2019][2805181.728219] LustreError: 118805:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128f0:0x99d5:0x0] [Wed Apr 10 22:27:20 2019][2805359.698027] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.60@o2ib4) [Wed Apr 10 22:27:20 2019][2805359.705526] Lustre: Skipped 1507 previous similar messages [Wed Apr 10 22:27:36 2019][2805375.732546] Lustre: fir-MDT0002: Client a11a9980-d36a-715f-c190-a6765cde5927 (at 10.8.1.22@o2ib6) reconnecting [Wed Apr 10 22:27:36 2019][2805375.742722] Lustre: Skipped 1499 previous similar messages [Wed Apr 10 22:37:21 2019][2805960.573475] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.35@o2ib4) [Wed Apr 10 22:37:21 2019][2805960.580975] Lustre: Skipped 1497 previous similar messages [Wed Apr 10 22:37:38 2019][2805977.512686] Lustre: fir-MDT0002: Client 83b8ff7e-e264-e3fb-e7eb-42bd933886ea (at 10.8.1.23@o2ib6) reconnecting [Wed Apr 10 22:37:38 2019][2805977.522865] Lustre: Skipped 1507 previous similar messages [Wed Apr 10 22:38:14 2019][2806013.282863] Lustre: MGS: haven't heard from client 4694839a-8554-10b4-73f9-fbcc10292ba8 (at 10.8.31.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b7de44400, cur 1554961094 expire 1554960944 last 1554960867 [Wed Apr 10 22:38:14 2019][2806013.303961] Lustre: Skipped 4 previous similar messages [Wed Apr 10 22:38:19 2019][2806018.255402] Lustre: fir-MDT0000: haven't heard from client b10270ab-2dff-3a94-2c78-6a892947d620 (at 10.8.31.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b558438e000, cur 1554961099 expire 1554960949 last 1554960872 [Wed Apr 10 22:38:22 2019][2806021.255392] Lustre: fir-MDT0002: haven't heard from client b10270ab-2dff-3a94-2c78-6a892947d620 (at 10.8.31.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b1e7d2000, cur 1554961102 expire 1554960952 last 1554960875 [Wed Apr 10 22:38:22 2019][2806021.277183] Lustre: Skipped 2 previous similar messages [Wed Apr 10 22:40:57 2019][2806176.952984] LustreError: 119107:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5aaa3cd100 x1628639188183024/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:3/0 lens 448/440 e 0 to 0 dl 1554961263 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 22:44:06 2019][2806365.958594] LustreError: 119248:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b880573d850 x1628639188986512/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:12/0 lens 448/440 e 0 to 0 dl 1554961452 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 22:44:27 2019][2806386.960831] LustreError: 119086:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b7cb6336050 x1628639189024336/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:3/0 lens 448/440 e 0 to 0 dl 1554961473 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 22:47:21 2019][2806560.730138] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Wed Apr 10 22:47:21 2019][2806560.737544] Lustre: Skipped 1518 previous similar messages [Wed Apr 10 22:47:38 2019][2806577.980943] Lustre: fir-MDT0002: Client bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83 (at 10.8.26.21@o2ib6) reconnecting [Wed Apr 10 22:47:38 2019][2806577.991211] Lustre: Skipped 1506 previous similar messages [Wed Apr 10 22:54:20 2019][2806979.274710] Lustre: fir-MDT0002: haven't heard from client bcdabe8d-f345-66f7-e967-2a74c6b4293a (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b87dd6d4800, cur 1554962060 expire 1554961910 last 1554961833 [Wed Apr 10 22:54:40 2019][2806999.275539] Lustre: fir-MDT0000: haven't heard from client bcdabe8d-f345-66f7-e967-2a74c6b4293a (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b59620eb800, cur 1554962080 expire 1554961930 last 1554961853 [Wed Apr 10 22:54:40 2019][2806999.297415] Lustre: Skipped 3 previous similar messages [Wed Apr 10 22:55:10 2019][2807029.560894] Lustre: 117464:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Wed Apr 10 22:55:10 2019][2807029.572811] Lustre: 117464:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Wed Apr 10 22:55:21 2019][2807040.601471] LustreError: 118811:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011927:0xb8ea:0x0] [Wed Apr 10 22:57:22 2019][2807161.953602] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Wed Apr 10 22:57:22 2019][2807161.963948] Lustre: Skipped 1498 previous similar messages [Wed Apr 10 22:57:39 2019][2807178.874784] Lustre: fir-MDT0002: Client 35ddd205-c174-6b95-20ef-c38a3193733f (at 10.9.108.35@o2ib4) reconnecting [Wed Apr 10 22:57:39 2019][2807178.885137] Lustre: Skipped 1500 previous similar messages [Wed Apr 10 22:59:07 2019][2807266.280387] Lustre: fir-MDT0000: haven't heard from client c2f72166-9dcf-6ceb-2251-8ebc7cc961d7 (at 10.9.102.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4d11654c00, cur 1554962347 expire 1554962197 last 1554962120 [Wed Apr 10 22:59:23 2019][2807282.281392] Lustre: fir-MDT0002: haven't heard from client c2f72166-9dcf-6ceb-2251-8ebc7cc961d7 (at 10.9.102.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7cb531e400, cur 1554962363 expire 1554962213 last 1554962136 [Wed Apr 10 22:59:23 2019][2807282.303382] Lustre: Skipped 3 previous similar messages [Wed Apr 10 23:00:54 2019][2807374.169538] LustreError: 119205:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6463b41800 x1628639191253760/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:0/0 lens 448/440 e 0 to 0 dl 1554962460 ref 1 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 23:01:11 2019][2807390.847827] Lustre: 118660:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Wed Apr 10 23:01:11 2019][2807390.847827] req@ffff8b60e4e48300 x1628745851469088/t146196931085(0) o36->eb030a79-9cda-5e09-c87b-4c55bbd9e3db@10.9.102.7@o2ib4:16/0 lens 512/2888 e 1 to 0 dl 1554962476 ref 2 fl Interpret:/0/0 rc 0/0 [Wed Apr 10 23:01:11 2019][2807390.877855] Lustre: 118660:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Wed Apr 10 23:02:26 2019][2807465.777322] LustreError: 118587:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1554962456, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b7101272880/0x53d0f216b535bb27 lrc: 3/1,0 mode: --/PR res: [0x2c0010764:0x11121:0x0].0x0 bits 0x13/0x8 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118587 timeout: 0 lvb_type: 0 [Wed Apr 10 23:02:26 2019][2807465.817151] LustreError: 118587:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Wed Apr 10 23:04:16 2019][2807575.983524] LNet: Service thread pid 118664 was inactive for 200.20s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Apr 10 23:04:16 2019][2807576.000726] LNet: Skipped 1 previous similar message [Wed Apr 10 23:04:16 2019][2807576.005874] Pid: 118664, comm: mdt03_077 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 23:04:16 2019][2807576.015877] Call Trace: [Wed Apr 10 23:04:16 2019][2807576.018526] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 23:04:16 2019][2807576.025663] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 23:04:16 2019][2807576.033041] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Wed Apr 10 23:04:16 2019][2807576.039973] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Wed Apr 10 23:04:16 2019][2807576.046639] [] mdt_reint_rec+0x83/0x210 [mdt] [Wed Apr 10 23:04:16 2019][2807576.052870] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Wed Apr 10 23:04:16 2019][2807576.059620] [] mdt_reint+0x67/0x140 [mdt] [Wed Apr 10 23:04:16 2019][2807576.065507] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 23:04:16 2019][2807576.072656] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 23:04:16 2019][2807576.080560] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 23:04:16 2019][2807576.087079] [] kthread+0xd1/0xe0 [Wed Apr 10 23:04:16 2019][2807576.092184] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 23:04:16 2019][2807576.098838] [] 0xffffffffffffffff [Wed Apr 10 23:04:16 2019][2807576.104051] LustreError: dumping log to /tmp/lustre-log.1554962656.118664 [Wed Apr 10 23:04:18 2019][2807577.541980] Pid: 118587, comm: mdt02_108 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Apr 10 23:04:18 2019][2807577.551980] Call Trace: [Wed Apr 10 23:04:18 2019][2807577.554630] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Apr 10 23:04:18 2019][2807577.561763] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Apr 10 23:04:18 2019][2807577.569162] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Apr 10 23:04:18 2019][2807577.576174] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Apr 10 23:04:18 2019][2807577.583372] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Wed Apr 10 23:04:18 2019][2807577.590049] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Wed Apr 10 23:04:18 2019][2807577.597238] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Apr 10 23:04:18 2019][2807577.604006] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Apr 10 23:04:18 2019][2807577.610686] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Apr 10 23:04:18 2019][2807577.617642] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Apr 10 23:04:18 2019][2807577.624953] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Apr 10 23:04:18 2019][2807577.631329] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Apr 10 23:04:18 2019][2807577.638447] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Apr 10 23:04:18 2019][2807577.646352] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Apr 10 23:04:18 2019][2807577.652870] [] kthread+0xd1/0xe0 [Wed Apr 10 23:04:18 2019][2807577.657980] [] ret_from_fork_nospec_begin+0xe/0x21 [Wed Apr 10 23:04:18 2019][2807577.664629] [] 0xffffffffffffffff [Wed Apr 10 23:07:23 2019][2807762.410121] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Wed Apr 10 23:07:23 2019][2807762.417434] Lustre: Skipped 1531 previous similar messages [Wed Apr 10 23:07:39 2019][2807778.993765] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Wed Apr 10 23:07:39 2019][2807779.004035] Lustre: Skipped 1528 previous similar messages [Wed Apr 10 23:15:47 2019][2808266.300424] Lustre: fir-MDT0002: haven't heard from client 9dc11af0-1d2b-c135-143e-e5a2cd09195e (at 10.8.19.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7414fc2800, cur 1554963347 expire 1554963197 last 1554963120 [Wed Apr 10 23:17:23 2019][2808362.463239] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.6@o2ib6) [Wed Apr 10 23:17:23 2019][2808362.470551] Lustre: Skipped 1534 previous similar messages [Wed Apr 10 23:17:40 2019][2808379.535068] Lustre: fir-MDT0002: Client e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6) reconnecting [Wed Apr 10 23:17:40 2019][2808379.545243] Lustre: Skipped 1523 previous similar messages [Wed Apr 10 23:24:51 2019][2808810.455535] Lustre: DEBUG MARKER: Wed Apr 10 23:24:51 2019 [Wed Apr 10 23:24:51 2019][2808810.455535] [Wed Apr 10 23:27:23 2019][2808962.750731] Lustre: fir-MDT0002: Connection restored to (at 10.9.103.11@o2ib4) [Wed Apr 10 23:27:23 2019][2808962.758228] Lustre: Skipped 1529 previous similar messages [Wed Apr 10 23:27:41 2019][2808980.991701] Lustre: fir-MDT0002: Client 83b8ff7e-e264-e3fb-e7eb-42bd933886ea (at 10.8.1.23@o2ib6) reconnecting [Wed Apr 10 23:27:41 2019][2808981.001878] Lustre: Skipped 1537 previous similar messages [Wed Apr 10 23:27:46 2019][2808985.775707] LNetError: 117119:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Wed Apr 10 23:37:23 2019][2809562.968965] Lustre: fir-MDT0002: Connection restored to 48747558-7da6-0440-a86c-cce04c28222a (at 10.8.7.22@o2ib6) [Wed Apr 10 23:37:23 2019][2809562.979416] Lustre: Skipped 1543 previous similar messages [Wed Apr 10 23:37:42 2019][2809581.516704] Lustre: fir-MDT0002: Client bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83 (at 10.8.26.21@o2ib6) reconnecting [Wed Apr 10 23:37:42 2019][2809581.526975] Lustre: Skipped 1536 previous similar messages [Wed Apr 10 23:40:31 2019][2809750.330350] Lustre: fir-MDT0000: haven't heard from client e133d3ba-741b-8aba-d6d6-045d6382599c (at 10.9.108.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4cc9c68400, cur 1554964831 expire 1554964681 last 1554964604 [Wed Apr 10 23:40:31 2019][2809750.352316] Lustre: Skipped 4 previous similar messages [Wed Apr 10 23:47:23 2019][2810163.091485] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.28@o2ib6) [Wed Apr 10 23:47:23 2019][2810163.098893] Lustre: Skipped 1531 previous similar messages [Wed Apr 10 23:47:43 2019][2810182.606220] Lustre: fir-MDT0002: Client 35ddd205-c174-6b95-20ef-c38a3193733f (at 10.9.108.35@o2ib4) reconnecting [Wed Apr 10 23:47:43 2019][2810182.616572] Lustre: Skipped 1528 previous similar messages [Wed Apr 10 23:57:26 2019][2810766.260474] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Wed Apr 10 23:57:26 2019][2810766.267799] Lustre: Skipped 1532 previous similar messages [Wed Apr 10 23:57:43 2019][2810782.730681] Lustre: fir-MDT0002: Client f53d399b-a6b5-36d0-034f-8c8c0efbb65f (at 10.9.106.11@o2ib4) reconnecting [Wed Apr 10 23:57:43 2019][2810782.741035] Lustre: Skipped 1540 previous similar messages [Thu Apr 11 00:00:46 2019][2810965.691782] LustreError: 119195:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b71eef6d400 x1628639198867600/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:22/0 lens 448/440 e 0 to 0 dl 1554966052 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 00:07:26 2019][2811366.321050] Lustre: fir-MDT0002: Connection restored to 3dde01db-8bab-e6a5-6af0-a17c56d94df2 (at 10.9.101.66@o2ib4) [Thu Apr 11 00:07:26 2019][2811366.331659] Lustre: Skipped 1544 previous similar messages [Thu Apr 11 00:07:44 2019][2811383.566784] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Thu Apr 11 00:07:44 2019][2811383.576900] Lustre: Skipped 1524 previous similar messages [Thu Apr 11 00:11:58 2019][2811637.538442] LustreError: 118806:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00109ae:0x19b7:0x0] [Thu Apr 11 00:17:26 2019][2811966.337373] Lustre: fir-MDT0002: Connection restored to (at 10.9.103.11@o2ib4) [Thu Apr 11 00:17:26 2019][2811966.344867] Lustre: Skipped 1526 previous similar messages [Thu Apr 11 00:17:45 2019][2811984.450031] Lustre: fir-MDT0002: Client 6d84db07-946f-3a55-fea2-55904c526e1b (at 10.8.1.21@o2ib6) reconnecting [Thu Apr 11 00:17:45 2019][2811984.460211] Lustre: Skipped 1533 previous similar messages [Thu Apr 11 00:26:40 2019][2812519.872248] LustreError: 119087:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b68f1e84450 x1628639202805088/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:16/0 lens 448/440 e 0 to 0 dl 1554967606 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 00:27:28 2019][2812567.366547] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Thu Apr 11 00:27:28 2019][2812567.376982] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 00:27:45 2019][2812584.601730] Lustre: fir-MDT0002: Client 931a3723-3247-0feb-2066-0d2fae060ca7 (at 10.9.101.66@o2ib4) reconnecting [Thu Apr 11 00:27:45 2019][2812584.612078] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 00:37:17 2019][2813156.399016] Lustre: fir-MDT0000: haven't heard from client aa0f3068-7b29-4f2f-44a4-ae65d30ca459 (at 10.9.113.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5c8c77e800, cur 1554968237 expire 1554968087 last 1554968010 [Thu Apr 11 00:37:17 2019][2813156.420895] Lustre: Skipped 4 previous similar messages [Thu Apr 11 00:37:26 2019][2813165.413274] Lustre: fir-MDT0003: haven't heard from client aa0f3068-7b29-4f2f-44a4-ae65d30ca459 (at 10.9.113.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b647afadc00, cur 1554968246 expire 1554968096 last 1554968019 [Thu Apr 11 00:37:26 2019][2813165.435151] Lustre: Skipped 3 previous similar messages [Thu Apr 11 00:37:28 2019][2813167.449878] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.24@o2ib4) [Thu Apr 11 00:37:28 2019][2813167.457372] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 00:37:45 2019][2813184.627326] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Thu Apr 11 00:37:45 2019][2813184.637683] Lustre: Skipped 1523 previous similar messages [Thu Apr 11 00:39:04 2019][2813264.053125] LNetError: 117122:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 00:40:01 2019][2813321.090292] Lustre: 118448:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554968394/real 1554968394] req@ffff8b71f67ce900 x1630439524275920/t0(0) o106->fir-MDT0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554968401 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Thu Apr 11 00:40:01 2019][2813321.117912] Lustre: 118448:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Thu Apr 11 00:40:04 2019][2813323.694353] Lustre: 118257:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554968397/real 1554968397] req@ffff8b5075eaf800 x1630439524335296/t0(0) o106->fir-MDT0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554968404 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Thu Apr 11 00:40:08 2019][2813328.102439] Lustre: 118312:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554968401/real 1554968401] req@ffff8b773a257500 x1630439524276400/t0(0) o106->fir-MDT0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554968408 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 00:40:08 2019][2813328.130041] Lustre: 118312:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Thu Apr 11 00:40:09 2019][2813329.127458] Lustre: 118195:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 00:40:09 2019][2813329.127458] req@ffff8b6457eb9e00 x1630279645108832/t0(0) o101->ac95d228-240d-517f-782b-313c0bf3e0aa@10.9.112.15@o2ib4:14/0 lens 480/568 e 1 to 0 dl 1554968414 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 00:40:09 2019][2813329.156617] Lustre: 118195:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 00:40:15 2019][2813335.128575] Lustre: 118448:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554968408/real 1554968408] req@ffff8b71f67ce900 x1630439524275920/t0(0) o106->fir-MDT0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554968415 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 00:40:15 2019][2813335.156190] Lustre: 118448:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Thu Apr 11 00:40:19 2019][2813338.537651] Lustre: 118447:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 00:40:19 2019][2813338.537651] req@ffff8b71ff3b8900 x1630386488193040/t0(0) o101->8795169d-5044-1420-66a3-7c635a142bc2@10.9.112.14@o2ib4:24/0 lens 480/568 e 0 to 0 dl 1554968424 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 00:40:25 2019][2813344.721778] Lustre: 118257:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554968418/real 1554968418] req@ffff8b5075eaf800 x1630439524335296/t0(0) o106->fir-MDT0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554968425 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 00:40:25 2019][2813344.749416] Lustre: 118257:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [Thu Apr 11 00:40:43 2019][2813363.140149] Lustre: 118312:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554968436/real 1554968436] req@ffff8b773a257500 x1630439524276400/t0(0) o106->fir-MDT0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554968443 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 00:40:43 2019][2813363.167751] Lustre: 118312:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [Thu Apr 11 00:41:18 2019][2813398.168850] Lustre: 118448:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554968471/real 1554968471] req@ffff8b71f67ce900 x1630439524275920/t0(0) o106->fir-MDT0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554968478 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 00:41:18 2019][2813398.196478] Lustre: 118448:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [Thu Apr 11 00:42:24 2019][2813463.764178] Lustre: 118257:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554968537/real 1554968537] req@ffff8b5075eaf800 x1630439524335296/t0(0) o106->fir-MDT0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1554968544 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 00:42:24 2019][2813463.791806] Lustre: 118257:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [Thu Apr 11 00:42:42 2019][2813481.411930] Lustre: fir-MDT0002: haven't heard from client 04a140c3-a482-7014-c768-6a1fe4cf0c6b (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7b9f24f000, cur 1554968562 expire 1554968412 last 1554968335 [Thu Apr 11 00:42:52 2019][2813491.405730] Lustre: fir-MDT0000: haven't heard from client 04a140c3-a482-7014-c768-6a1fe4cf0c6b (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8be46d9c00, cur 1554968572 expire 1554968422 last 1554968345 [Thu Apr 11 00:42:52 2019][2813491.427697] Lustre: Skipped 3 previous similar messages [Thu Apr 11 00:42:52 2019][2813491.433194] Lustre: 118257:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (58:117s); client may timeout. req@ffff8b5556413600 x1630386488216464/t0(0) o101->8795169d-5044-1420-66a3-7c635a142bc2@10.9.112.14@o2ib4:27/0 lens 480/536 e 0 to 0 dl 1554968455 ref 1 fl Complete:/0/0 rc 301/301 [Thu Apr 11 00:42:52 2019][2813491.462289] Lustre: 118257:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1 previous similar message [Thu Apr 11 00:43:38 2019][2813538.260566] LustreError: 118951:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001117f:0x1731:0x0] [Thu Apr 11 00:45:49 2019][2813668.409288] Lustre: fir-MDT0001: haven't heard from client 24bebcd7-07c2-f8f4-f94f-ff1d34c61e7c (at 10.8.13.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b514841e000, cur 1554968749 expire 1554968599 last 1554968522 [Thu Apr 11 00:47:30 2019][2813769.732064] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Thu Apr 11 00:47:30 2019][2813769.739381] Lustre: Skipped 1541 previous similar messages [Thu Apr 11 00:47:46 2019][2813785.563926] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Thu Apr 11 00:47:46 2019][2813785.574109] Lustre: Skipped 1551 previous similar messages [Thu Apr 11 00:50:10 2019][2813930.191439] Lustre: 118421:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Thu Apr 11 00:54:19 2019][2814179.114760] LustreError: 118535:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8567f68c00 x1628639207051536/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:25/0 lens 448/440 e 0 to 0 dl 1554969265 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 00:57:30 2019][2814370.011243] Lustre: fir-MDT0002: Connection restored to 3dde01db-8bab-e6a5-6af0-a17c56d94df2 (at 10.9.101.66@o2ib4) [Thu Apr 11 00:57:30 2019][2814370.022042] Lustre: Skipped 1535 previous similar messages [Thu Apr 11 00:57:46 2019][2814386.264058] Lustre: fir-MDT0002: Client 26c07b92-3e88-a582-abe6-28f16a4cf62b (at 10.9.108.26@o2ib4) reconnecting [Thu Apr 11 00:57:46 2019][2814386.274455] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 01:00:16 2019][2814536.184776] LustreError: 119142:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6aedcc5100 x1628639207675184/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:22/0 lens 448/440 e 0 to 0 dl 1554969622 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 01:07:30 2019][2814970.106106] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.19@o2ib4) [Thu Apr 11 01:07:30 2019][2814970.113595] Lustre: Skipped 1538 previous similar messages [Thu Apr 11 01:07:47 2019][2814986.514779] Lustre: fir-MDT0002: Client e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6) reconnecting [Thu Apr 11 01:07:47 2019][2814986.524956] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 01:09:22 2019][2815082.285499] LustreError: 118085:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b7a5ca98300 x1628639208639424/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:28/0 lens 448/440 e 0 to 0 dl 1554970168 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 01:13:55 2019][2815355.299577] LustreError: 119142:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b665ea7a100 x1628639209116416/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:1/0 lens 448/440 e 0 to 0 dl 1554970441 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 01:17:31 2019][2815570.826858] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Thu Apr 11 01:17:31 2019][2815570.837293] Lustre: Skipped 1539 previous similar messages [Thu Apr 11 01:17:47 2019][2815586.542549] Lustre: fir-MDT0002: Client 3c1359c4-6f8a-28e8-3fe1-a0a63b3ad896 (at 10.9.102.1@o2ib4) reconnecting [Thu Apr 11 01:17:47 2019][2815586.552813] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 01:27:31 2019][2816170.899905] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.19@o2ib6) [Thu Apr 11 01:27:31 2019][2816170.907308] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 01:27:47 2019][2816186.655999] Lustre: fir-MDT0002: Client f5140abb-0b33-a9a1-9459-14bf253ba61e (at 10.9.106.60@o2ib4) reconnecting [Thu Apr 11 01:27:47 2019][2816186.666348] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 01:37:32 2019][2816771.909390] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.18@o2ib6) [Thu Apr 11 01:37:32 2019][2816771.916708] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 01:37:47 2019][2816787.102255] Lustre: fir-MDT0002: Client 48186cb7-0668-1018-7ab1-1555c19d856e (at 10.9.107.23@o2ib4) reconnecting [Thu Apr 11 01:37:47 2019][2816787.112603] Lustre: Skipped 1534 previous similar messages [Thu Apr 11 01:47:32 2019][2817372.249548] Lustre: fir-MDT0002: Connection restored to 15562cac-eb22-f844-7b88-546ba9ddee38 (at 10.9.101.71@o2ib4) [Thu Apr 11 01:47:32 2019][2817372.260156] Lustre: Skipped 1530 previous similar messages [Thu Apr 11 01:47:47 2019][2817387.301334] Lustre: fir-MDT0002: Client 35ddd205-c174-6b95-20ef-c38a3193733f (at 10.9.108.35@o2ib4) reconnecting [Thu Apr 11 01:47:47 2019][2817387.311684] Lustre: Skipped 1530 previous similar messages [Thu Apr 11 01:57:32 2019][2817972.283981] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Thu Apr 11 01:57:32 2019][2817972.291387] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 01:57:48 2019][2817988.095428] Lustre: fir-MDT0002: Client 70db6507-827a-2e40-1c44-099543d1512c (at 10.9.108.33@o2ib4) reconnecting [Thu Apr 11 01:57:48 2019][2817988.105778] Lustre: Skipped 1534 previous similar messages [Thu Apr 11 02:05:49 2019][2818469.259257] Lustre: 118384:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554973542/real 1554973542] req@ffff8b79473d1b00 x1630439635419632/t0(0) o106->fir-MDT0003@10.8.10.30@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554973549 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Thu Apr 11 02:05:49 2019][2818469.286793] Lustre: 118384:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [Thu Apr 11 02:05:57 2019][2818477.301416] Lustre: 118572:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 02:05:57 2019][2818477.301416] req@ffff8b6f56bf6300 x1629249044775616/t0(0) o101->bafe047a-89ea-21d5-a24e-5c407648cfc7@10.0.10.3@o2ib7:2/0 lens 480/568 e 1 to 0 dl 1554973562 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 02:05:57 2019][2818477.330316] Lustre: 118572:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 02:06:10 2019][2818490.276681] Lustre: 118439:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554973563/real 1554973563] req@ffff8b4d192b3300 x1630439635419760/t0(0) o106->fir-MDT0003@10.8.10.30@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1554973570 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 02:06:10 2019][2818490.304217] Lustre: 118439:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [Thu Apr 11 02:06:26 2019][2818505.507105] Lustre: fir-MDT0000: haven't heard from client 6bd95fb2-b79e-c4d4-f439-2ca26824ec37 (at 10.8.10.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b52b42ef800, cur 1554973586 expire 1554973436 last 1554973359 [Thu Apr 11 02:06:26 2019][2818505.528977] Lustre: Skipped 4 previous similar messages [Thu Apr 11 02:06:35 2019][2818515.276183] Lustre: 118408:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 02:06:35 2019][2818515.276183] req@ffff8b7921718f00 x1628742029470608/t0(0) o101->ba312893-3b9e-ec41-cd1b-c94457862133@10.9.101.16@o2ib4:10/0 lens 584/3264 e 1 to 0 dl 1554973600 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 02:06:35 2019][2818515.305456] Lustre: 118408:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 02:06:45 2019][2818524.574369] Lustre: 118622:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 02:06:45 2019][2818524.574369] req@ffff8b8b06348c00 x1628578176363136/t0(0) o101->bf1d2379-cb89-95cb-59cc-fc2243614630@10.9.108.49@o2ib4:20/0 lens 584/3264 e 0 to 0 dl 1554973610 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 02:06:48 2019][2818528.499452] Lustre: 118678:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1554973601/real 1554973601] req@ffff8b8b10328300 x1630439635703312/t0(0) o104->fir-MDT0001@10.9.101.9@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1554973608 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 02:06:49 2019][2818528.526987] Lustre: 118678:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [Thu Apr 11 02:07:29 2019][2818568.775549] LustreError: 119089:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b4f56382450 x1628639214819168/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:5/0 lens 448/440 e 0 to 0 dl 1554973655 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 02:07:33 2019][2818572.748644] Lustre: fir-MDT0002: Connection restored to 48747558-7da6-0440-a86c-cce04c28222a (at 10.8.7.22@o2ib6) [Thu Apr 11 02:07:33 2019][2818572.759101] Lustre: Skipped 1546 previous similar messages [Thu Apr 11 02:07:42 2019][2818581.508600] Lustre: fir-MDT0002: haven't heard from client ca9e0fa4-34eb-19b9-7e44-5f8f37e75cea (at 10.9.101.9@o2ib4) in 174 seconds. I think it's dead, and I am evicting it. exp ffff8b763abed400, cur 1554973662 expire 1554973512 last 1554973488 [Thu Apr 11 02:07:42 2019][2818581.530519] Lustre: Skipped 4 previous similar messages [Thu Apr 11 02:07:48 2019][2818588.368354] Lustre: fir-MDT0002: Client 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4) reconnecting [Thu Apr 11 02:07:48 2019][2818588.378704] Lustre: Skipped 1538 previous similar messages [Thu Apr 11 02:08:35 2019][2818634.515011] Lustre: fir-MDT0003: haven't heard from client ca9e0fa4-34eb-19b9-7e44-5f8f37e75cea (at 10.9.101.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6059725400, cur 1554973715 expire 1554973565 last 1554973488 [Thu Apr 11 02:08:35 2019][2818634.536916] Lustre: Skipped 2 previous similar messages [Thu Apr 11 02:17:33 2019][2819172.853339] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.12@o2ib6) [Thu Apr 11 02:17:33 2019][2819172.860658] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 02:17:49 2019][2819188.583340] Lustre: fir-MDT0002: Client 83b8ff7e-e264-e3fb-e7eb-42bd933886ea (at 10.8.1.23@o2ib6) reconnecting [Thu Apr 11 02:17:49 2019][2819188.593517] Lustre: Skipped 1533 previous similar messages [Thu Apr 11 02:27:33 2019][2819773.423913] Lustre: fir-MDT0002: Connection restored to 30d4579a-fbb3-061d-f340-d61cdf6fe8c4 (at 10.9.108.33@o2ib4) [Thu Apr 11 02:27:33 2019][2819773.434526] Lustre: Skipped 1538 previous similar messages [Thu Apr 11 02:27:49 2019][2819788.667363] Lustre: fir-MDT0002: Client 10601291-a183-8deb-41a7-af6620223334 (at 10.9.108.36@o2ib4) reconnecting [Thu Apr 11 02:27:49 2019][2819788.677720] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 02:37:33 2019][2820373.459247] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.18@o2ib4) [Thu Apr 11 02:37:33 2019][2820373.466745] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 02:37:49 2019][2820389.056258] Lustre: fir-MDT0002: Client 837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579 (at 10.9.104.22@o2ib4) reconnecting [Thu Apr 11 02:37:49 2019][2820389.066605] Lustre: Skipped 1533 previous similar messages [Thu Apr 11 02:40:27 2019][2820547.113248] Lustre: 118160:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 02:47:34 2019][2820973.794958] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Thu Apr 11 02:47:34 2019][2820973.802269] Lustre: Skipped 1535 previous similar messages [Thu Apr 11 02:47:50 2019][2820989.631873] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Thu Apr 11 02:47:50 2019][2820989.642046] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 02:51:28 2019][2821207.561634] Lustre: fir-MDT0000: haven't heard from client fa55c52c-0628-34e2-e09b-9c3f85d3d952 (at 10.8.25.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b504c1cb400, cur 1554976288 expire 1554976138 last 1554976061 [Thu Apr 11 02:51:28 2019][2821207.583518] Lustre: Skipped 1 previous similar message [Thu Apr 11 02:51:41 2019][2821220.563565] Lustre: MGS: haven't heard from client 69737854-d82c-93ca-b58d-0abfb9f474f0 (at 10.8.25.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8b7de42800, cur 1554976301 expire 1554976151 last 1554976074 [Thu Apr 11 02:51:41 2019][2821220.584756] Lustre: Skipped 7 previous similar messages [Thu Apr 11 02:57:34 2019][2821574.026141] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.36@o2ib4) [Thu Apr 11 02:57:34 2019][2821574.033626] Lustre: Skipped 1533 previous similar messages [Thu Apr 11 02:57:50 2019][2821589.946942] Lustre: fir-MDT0002: Client f4b95b51-aa49-8c42-0e74-2b9d33fd9533 (at 10.9.101.64@o2ib4) reconnecting [Thu Apr 11 02:57:50 2019][2821589.957291] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 03:01:44 2019][2821824.262230] Lustre: 118251:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:07:34 2019][2822174.410936] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.22@o2ib4) [Thu Apr 11 03:07:34 2019][2822174.418431] Lustre: Skipped 1526 previous similar messages [Thu Apr 11 03:07:52 2019][2822192.025974] Lustre: fir-MDT0002: Client 6d84db07-946f-3a55-fea2-55904c526e1b (at 10.8.1.21@o2ib6) reconnecting [Thu Apr 11 03:07:52 2019][2822192.036156] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 03:12:39 2019][2822478.985571] Lustre: 117465:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:12:39 2019][2822478.997487] Lustre: 117465:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Thu Apr 11 03:17:34 2019][2822774.434809] Lustre: fir-MDT0002: Connection restored to 6e57a6e6-07ae-4b36-8d72-608075772c63 (at 10.9.106.55@o2ib4) [Thu Apr 11 03:17:34 2019][2822774.445424] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 03:17:38 2019][2822777.703441] Lustre: 118286:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:17:38 2019][2822777.715351] Lustre: 118286:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Thu Apr 11 03:17:52 2019][2822792.262408] Lustre: fir-MDT0002: Client 10601291-a183-8deb-41a7-af6620223334 (at 10.9.108.36@o2ib4) reconnecting [Thu Apr 11 03:17:52 2019][2822792.272759] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 03:19:23 2019][2822883.224659] Lustre: 118441:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:19:23 2019][2822883.236587] Lustre: 118441:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Thu Apr 11 03:21:31 2019][2823011.329027] Lustre: 118363:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:25:39 2019][2823258.742663] Lustre: 118286:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:25:39 2019][2823258.754733] Lustre: 118286:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 13 previous similar messages [Thu Apr 11 03:27:34 2019][2823374.465064] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.28@o2ib6) [Thu Apr 11 03:27:34 2019][2823374.472467] Lustre: Skipped 1538 previous similar messages [Thu Apr 11 03:27:53 2019][2823392.627248] Lustre: fir-MDT0002: Client 837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579 (at 10.9.104.22@o2ib4) reconnecting [Thu Apr 11 03:27:53 2019][2823392.637600] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 03:33:32 2019][2823732.150755] Lustre: 118396:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:35:00 2019][2823819.602915] LustreError: 119240:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b835bb0d100 x1628639224209728/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:5/0 lens 448/440 e 0 to 0 dl 1554978905 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 03:37:35 2019][2823974.987579] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.24@o2ib6) [Thu Apr 11 03:37:35 2019][2823974.994907] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 03:37:39 2019][2823978.785309] Lustre: 118286:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:37:39 2019][2823978.797240] Lustre: 118286:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Thu Apr 11 03:37:53 2019][2823992.721043] Lustre: fir-MDT0002: Client 1bde1f20-37f1-ed48-746c-38acfc0e9a71 (at 10.9.106.55@o2ib4) reconnecting [Thu Apr 11 03:37:53 2019][2823992.731395] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 03:40:47 2019][2824167.170227] Lustre: 118211:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:40:47 2019][2824167.182136] Lustre: 118211:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 18 previous similar messages [Thu Apr 11 03:44:27 2019][2824386.689257] LustreError: 119113:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6b1637d400 x1628639225223840/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:3/0 lens 448/440 e 0 to 0 dl 1554979473 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 03:47:36 2019][2824575.695789] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.34@o2ib4) [Thu Apr 11 03:47:36 2019][2824575.703277] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 03:47:53 2019][2824593.127979] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Thu Apr 11 03:47:53 2019][2824593.138327] Lustre: Skipped 1524 previous similar messages [Thu Apr 11 03:49:26 2019][2824686.026932] Lustre: 118414:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:49:26 2019][2824686.038840] Lustre: 118414:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 12 previous similar messages [Thu Apr 11 03:51:41 2019][2824820.660798] Lustre: 118441:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:51:41 2019][2824820.672708] Lustre: 118441:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 33 previous similar messages [Thu Apr 11 03:56:38 2019][2825118.336346] Lustre: 118548:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 03:56:38 2019][2825118.348259] Lustre: 118548:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Thu Apr 11 03:57:36 2019][2825175.821941] Lustre: fir-MDT0002: Connection restored to 98f9d97b-94c5-a227-0ea6-97af7ec1ed2e (at 10.9.104.17@o2ib4) [Thu Apr 11 03:57:36 2019][2825175.832553] Lustre: Skipped 1530 previous similar messages [Thu Apr 11 03:57:53 2019][2825193.158124] Lustre: fir-MDT0002: Client 94d4bb8d-4025-4bb2-3e79-daa06ff1054f (at 10.9.108.15@o2ib4) reconnecting [Thu Apr 11 03:57:53 2019][2825193.168475] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 04:07:33 2019][2825772.897518] LustreError: 118082:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b4d18525100 x1628639227752784/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:9/0 lens 448/440 e 0 to 0 dl 1554980859 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 04:07:36 2019][2825776.168568] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Thu Apr 11 04:07:36 2019][2825776.175887] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 04:07:54 2019][2825793.897735] Lustre: fir-MDT0002: Client 01f19ed0-770e-84b4-7586-12f4195b7925 (at 10.9.108.34@o2ib4) reconnecting [Thu Apr 11 04:07:54 2019][2825793.908078] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 04:10:00 2019][2825919.923092] LustreError: 119191:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b75923e4b00 x1628639228026448/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:6/0 lens 448/440 e 0 to 0 dl 1554981006 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 04:17:36 2019][2826376.425264] Lustre: fir-MDT0002: Connection restored to (at 10.9.114.5@o2ib4) [Thu Apr 11 04:17:36 2019][2826376.432666] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 04:17:54 2019][2826393.963688] Lustre: fir-MDT0002: Client 19decd65-35ff-8baa-0691-542685855960 (at 10.8.27.9@o2ib6) reconnecting [Thu Apr 11 04:17:54 2019][2826393.973869] Lustre: Skipped 1530 previous similar messages [Thu Apr 11 04:27:37 2019][2826976.860491] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Thu Apr 11 04:27:37 2019][2826976.867909] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 04:27:54 2019][2826993.979868] Lustre: fir-MDT0002: Client eb030a79-9cda-5e09-c87b-4c55bbd9e3db (at 10.9.102.7@o2ib4) reconnecting [Thu Apr 11 04:27:54 2019][2826993.990134] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 04:33:48 2019][2827348.202651] LustreError: 118501:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b5179336f00 x1628639230638032/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:24/0 lens 448/440 e 0 to 0 dl 1554982434 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 04:37:37 2019][2827576.910477] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.18@o2ib4) [Thu Apr 11 04:37:37 2019][2827576.917965] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 04:37:54 2019][2827594.179998] Lustre: fir-MDT0002: Client 369e48ad-81ca-5881-14c8-c80031c22f3c (at 10.9.104.17@o2ib4) reconnecting [Thu Apr 11 04:37:54 2019][2827594.190346] Lustre: Skipped 1526 previous similar messages [Thu Apr 11 04:38:49 2019][2827649.284839] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 04:47:38 2019][2828177.872245] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Thu Apr 11 04:47:38 2019][2828177.879566] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 04:47:54 2019][2828194.480708] Lustre: fir-MDT0002: Client e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6) reconnecting [Thu Apr 11 04:47:54 2019][2828194.490888] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 04:49:11 2019][2828271.518157] Lustre: 118160:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0003: Failure to clear the changelog for user 1: -22 [Thu Apr 11 04:49:11 2019][2828271.530071] Lustre: 118160:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 14 previous similar messages [Thu Apr 11 04:56:33 2019][2828713.403842] LustreError: 118528:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b834c707800 x1628639233075536/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:9/0 lens 448/440 e 0 to 0 dl 1554983799 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 04:57:38 2019][2828777.943343] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.1@o2ib4) [Thu Apr 11 04:57:38 2019][2828777.950752] Lustre: Skipped 1528 previous similar messages [Thu Apr 11 04:57:55 2019][2828794.993359] Lustre: fir-MDT0002: Client 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4) reconnecting [Thu Apr 11 04:57:55 2019][2828795.003704] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 05:07:38 2019][2829378.598679] Lustre: fir-MDT0002: Connection restored to 1f1457dd-8be9-0520-54b3-f55bdd85996c (at 10.8.7.31@o2ib6) [Thu Apr 11 05:07:38 2019][2829378.609118] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 05:07:56 2019][2829396.052258] Lustre: fir-MDT0002: Client 6d84db07-946f-3a55-fea2-55904c526e1b (at 10.8.1.21@o2ib6) reconnecting [Thu Apr 11 05:07:56 2019][2829396.062439] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 05:17:39 2019][2829978.876431] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Thu Apr 11 05:17:39 2019][2829978.886867] Lustre: Skipped 1538 previous similar messages [Thu Apr 11 05:17:56 2019][2829996.150475] Lustre: fir-MDT0002: Client 3c1359c4-6f8a-28e8-3fe1-a0a63b3ad896 (at 10.9.102.1@o2ib4) reconnecting [Thu Apr 11 05:17:56 2019][2829996.160736] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 05:26:39 2019][2830519.660126] LustreError: 119017:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b774de66000 x1628639236285968/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:15/0 lens 448/440 e 0 to 0 dl 1554985605 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 05:27:39 2019][2830579.055629] Lustre: fir-MDT0002: Connection restored to (at 10.8.18.30@o2ib6) [Thu Apr 11 05:27:39 2019][2830579.063032] Lustre: Skipped 1523 previous similar messages [Thu Apr 11 05:27:56 2019][2830596.625665] Lustre: fir-MDT0002: Client 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4) reconnecting [Thu Apr 11 05:27:56 2019][2830596.636022] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 05:37:36 2019][2831175.759957] Lustre: fir-MDT0000: haven't heard from client 1acf9c42-ef2d-69bb-682d-4ba76391625a (at 10.8.9.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b88fca04400, cur 1554986256 expire 1554986106 last 1554986029 [Thu Apr 11 05:37:36 2019][2831175.781691] Lustre: Skipped 1 previous similar message [Thu Apr 11 05:37:40 2019][2831180.709398] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.23@o2ib4) [Thu Apr 11 05:37:40 2019][2831180.716891] Lustre: Skipped 1539 previous similar messages [Thu Apr 11 05:37:56 2019][2831196.645042] Lustre: fir-MDT0002: Client 58899832-571e-4a18-664b-6f5c1308e27e (at 10.8.18.18@o2ib6) reconnecting [Thu Apr 11 05:37:56 2019][2831196.655304] Lustre: Skipped 1533 previous similar messages [Thu Apr 11 05:47:41 2019][2831780.807147] Lustre: fir-MDT0002: Connection restored to 15562cac-eb22-f844-7b88-546ba9ddee38 (at 10.9.101.71@o2ib4) [Thu Apr 11 05:47:41 2019][2831780.817758] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 05:47:57 2019][2831796.858060] Lustre: fir-MDT0002: Client 35ddd205-c174-6b95-20ef-c38a3193733f (at 10.9.108.35@o2ib4) reconnecting [Thu Apr 11 05:47:57 2019][2831796.868439] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 05:57:41 2019][2832380.926047] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.22@o2ib6) [Thu Apr 11 05:57:41 2019][2832380.933365] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 05:57:58 2019][2832398.264800] Lustre: fir-MDT0002: Client e18a93c6-a11b-0b52-af65-1483de6beb82 (at 10.8.8.18@o2ib6) reconnecting [Thu Apr 11 05:57:58 2019][2832398.274988] Lustre: Skipped 1538 previous similar messages [Thu Apr 11 06:05:52 2019][2832872.016662] LustreError: 117476:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b4ce0440300 x1628639240292688/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:28/0 lens 448/440 e 0 to 0 dl 1554987958 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 06:07:42 2019][2832982.044604] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 06:07:42 2019][2832982.055125] Lustre: Skipped 1544 previous similar messages [Thu Apr 11 06:07:59 2019][2832999.022368] Lustre: fir-MDT0002: Client 99abaf9a-6d16-bfe7-2d5c-9e199780b8b7 (at 10.9.101.71@o2ib4) reconnecting [Thu Apr 11 06:07:59 2019][2832999.032724] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 06:09:43 2019][2833103.019216] LustreError: 119262:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b773a251200 x1628639240707328/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:19/0 lens 448/440 e 0 to 0 dl 1554988189 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 06:11:49 2019][2833229.022395] LustreError: 119113:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6019700300 x1628639240932800/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:25/0 lens 448/440 e 0 to 0 dl 1554988315 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 06:13:37 2019][2833336.802563] Lustre: fir-MDT0000: haven't heard from client f6a53bd9-6f03-546e-1945-491ccd12b33c (at 10.9.112.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b58446c7000, cur 1554988417 expire 1554988267 last 1554988190 [Thu Apr 11 06:13:37 2019][2833336.824437] Lustre: Skipped 4 previous similar messages [Thu Apr 11 06:16:22 2019][2833502.109054] LustreError: 118570:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8338fb0000 x1628639241425712/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:28/0 lens 448/440 e 0 to 0 dl 1554988588 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 06:17:42 2019][2833582.295654] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.28@o2ib6) [Thu Apr 11 06:17:42 2019][2833582.303053] Lustre: Skipped 1525 previous similar messages [Thu Apr 11 06:17:59 2019][2833599.082567] Lustre: fir-MDT0002: Client e4a8aa81-b0b9-5652-c269-3fdf6ff70c2f (at 10.9.102.23@o2ib4) reconnecting [Thu Apr 11 06:17:59 2019][2833599.092920] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 06:27:44 2019][2834184.288327] Lustre: fir-MDT0002: Connection restored to 1f1457dd-8be9-0520-54b3-f55bdd85996c (at 10.8.7.31@o2ib6) [Thu Apr 11 06:27:44 2019][2834184.298758] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 06:27:59 2019][2834199.204878] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Thu Apr 11 06:27:59 2019][2834199.215235] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 06:37:44 2019][2834784.337862] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.16@o2ib6) [Thu Apr 11 06:37:44 2019][2834784.345271] Lustre: Skipped 1530 previous similar messages [Thu Apr 11 06:37:59 2019][2834799.299268] Lustre: fir-MDT0002: Client a11a9980-d36a-715f-c190-a6765cde5927 (at 10.8.1.22@o2ib6) reconnecting [Thu Apr 11 06:37:59 2019][2834799.309454] Lustre: Skipped 1523 previous similar messages [Thu Apr 11 06:38:46 2019][2834846.297989] LustreError: 119201:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b710fe08600 x1628639243818352/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:22/0 lens 448/440 e 0 to 0 dl 1554989932 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 06:41:55 2019][2835035.304661] LustreError: 119126:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b4d1a4d5100 x1628639244151440/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:1/0 lens 448/440 e 0 to 0 dl 1554990121 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 06:47:44 2019][2835384.828750] Lustre: fir-MDT0002: Connection restored to (at 10.8.18.30@o2ib6) [Thu Apr 11 06:47:45 2019][2835384.836157] Lustre: Skipped 1533 previous similar messages [Thu Apr 11 06:47:59 2019][2835399.448429] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Thu Apr 11 06:47:59 2019][2835399.458696] Lustre: Skipped 1539 previous similar messages [Thu Apr 11 06:50:40 2019][2835560.383907] LustreError: 119241:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b5321226000 x1628639245093056/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:16/0 lens 448/440 e 0 to 0 dl 1554990646 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 06:52:01 2019][2835640.848522] Lustre: fir-MDT0001: haven't heard from client 1148df5e-3aca-0334-aa9c-09daaee4cb60 (at 10.8.13.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b52933dbc00, cur 1554990721 expire 1554990571 last 1554990494 [Thu Apr 11 06:52:01 2019][2835640.870398] Lustre: Skipped 4 previous similar messages [Thu Apr 11 06:54:10 2019][2835770.453288] LustreError: 118525:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b85a6670c00 x1628639245467392/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:16/0 lens 448/440 e 0 to 0 dl 1554990856 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 06:56:22 2019][2835901.853752] Lustre: fir-MDT0001: haven't heard from client 4bb7cf7b-5290-c622-1a9e-3a52b87105b0 (at 10.9.101.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b52f83b7800, cur 1554990982 expire 1554990832 last 1554990755 [Thu Apr 11 06:56:22 2019][2835901.875719] Lustre: Skipped 4 previous similar messages [Thu Apr 11 06:56:32 2019][2835911.854156] Lustre: fir-MDT0003: haven't heard from client 4bb7cf7b-5290-c622-1a9e-3a52b87105b0 (at 10.9.101.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5dde369400, cur 1554990992 expire 1554990842 last 1554990765 [Thu Apr 11 06:57:45 2019][2835985.614438] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 06:57:45 2019][2835985.624956] Lustre: Skipped 1541 previous similar messages [Thu Apr 11 06:58:00 2019][2836000.250164] Lustre: fir-MDT0002: Client 86cc2579-6592-d6be-ad47-10ff366dfbdf (at 10.8.8.19@o2ib6) reconnecting [Thu Apr 11 06:58:00 2019][2836000.260345] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 07:04:40 2019][2836400.542021] LustreError: 119080:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b79b2a90f00 x1628639246597824/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:16/0 lens 448/440 e 0 to 0 dl 1554991486 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 07:07:45 2019][2836585.798145] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.26@o2ib4) [Thu Apr 11 07:07:45 2019][2836585.805631] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 07:08:02 2019][2836602.066741] Lustre: fir-MDT0002: Client 1fed4b9d-ca5d-165d-887c-2f1234134d0c (at 10.8.9.10@o2ib6) reconnecting [Thu Apr 11 07:08:02 2019][2836602.076928] Lustre: Skipped 1533 previous similar messages [Thu Apr 11 07:09:55 2019][2836715.565980] LustreError: 119256:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b58daad3600 x1628639247166128/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:1/0 lens 448/440 e 0 to 0 dl 1554991801 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 07:17:46 2019][2837185.865526] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Thu Apr 11 07:17:46 2019][2837185.872844] Lustre: Skipped 1534 previous similar messages [Thu Apr 11 07:18:02 2019][2837202.745979] Lustre: fir-MDT0002: Client ee5f7d3d-8ffe-7ca2-e4f8-dd26c84b3cb6 (at 10.9.107.48@o2ib4) reconnecting [Thu Apr 11 07:18:02 2019][2837202.756340] Lustre: Skipped 1539 previous similar messages [Thu Apr 11 07:27:25 2019][2837765.766527] LustreError: 117902:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5f5f718600 x1628639249032912/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:1/0 lens 448/440 e 0 to 0 dl 1554992851 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 07:27:46 2019][2837786.767256] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.34@o2ib4) [Thu Apr 11 07:27:46 2019][2837786.774752] Lustre: Skipped 1541 previous similar messages [Thu Apr 11 07:28:03 2019][2837802.921069] Lustre: fir-MDT0002: Client 398eea62-82fb-c936-bc1a-b5cbed90bbf5 (at 10.8.8.12@o2ib6) reconnecting [Thu Apr 11 07:28:03 2019][2837802.931245] Lustre: Skipped 1521 previous similar messages [Thu Apr 11 07:37:47 2019][2838387.343161] Lustre: fir-MDT0002: Connection restored to 3f1d1974-7ed8-ffd0-1256-fba8ddd54f53 (at 10.8.9.10@o2ib6) [Thu Apr 11 07:37:47 2019][2838387.353600] Lustre: Skipped 1530 previous similar messages [Thu Apr 11 07:38:03 2019][2838403.023405] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Thu Apr 11 07:38:03 2019][2838403.033673] Lustre: Skipped 1541 previous similar messages [Thu Apr 11 07:40:22 2019][2838542.876593] LustreError: 119181:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b53c359bf00 x1628639250416576/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:28/0 lens 448/440 e 0 to 0 dl 1554993628 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 07:43:53 2019][2838752.886738] LustreError: 119101:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b501734f500 x1628639250790672/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:28/0 lens 448/440 e 0 to 0 dl 1554993838 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 07:47:47 2019][2838987.358353] Lustre: fir-MDT0002: Connection restored to 3dde01db-8bab-e6a5-6af0-a17c56d94df2 (at 10.9.101.66@o2ib4) [Thu Apr 11 07:47:47 2019][2838987.368961] Lustre: Skipped 1538 previous similar messages [Thu Apr 11 07:48:03 2019][2839003.766786] Lustre: fir-MDT0002: Client 86cc2579-6592-d6be-ad47-10ff366dfbdf (at 10.8.8.19@o2ib6) reconnecting [Thu Apr 11 07:48:03 2019][2839003.776961] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 07:56:08 2019][2839488.043250] LustreError: 119051:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6f8ba0d700 x1628639252111824/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:14/0 lens 448/440 e 0 to 0 dl 1554994574 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 07:57:47 2019][2839587.631173] Lustre: fir-MDT0002: Connection restored to (at 10.9.114.5@o2ib4) [Thu Apr 11 07:57:47 2019][2839587.638575] Lustre: Skipped 1524 previous similar messages [Thu Apr 11 07:58:04 2019][2839604.238152] Lustre: fir-MDT0002: Client e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6) reconnecting [Thu Apr 11 07:58:04 2019][2839604.248347] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 08:00:20 2019][2839740.077075] LustreError: 119208:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b8c29a8f850 x1628639252570352/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:26/0 lens 448/440 e 0 to 0 dl 1554994826 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 08:07:48 2019][2840188.370289] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Thu Apr 11 08:07:48 2019][2840188.377690] Lustre: Skipped 1535 previous similar messages [Thu Apr 11 08:08:04 2019][2840204.496302] Lustre: fir-MDT0002: Client 99b2b05a-80e4-edd2-5271-b136b8e4528d (at 10.9.108.67@o2ib4) reconnecting [Thu Apr 11 08:08:04 2019][2840204.506654] Lustre: Skipped 1535 previous similar messages [Thu Apr 11 08:14:20 2019][2840580.179669] LustreError: 119213:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b68f1e86050 x1628639254085472/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:26/0 lens 448/440 e 0 to 0 dl 1554995666 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 08:17:48 2019][2840788.422248] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.15@o2ib4) [Thu Apr 11 08:17:48 2019][2840788.429737] Lustre: Skipped 1530 previous similar messages [Thu Apr 11 08:18:04 2019][2840804.775573] Lustre: fir-MDT0002: Client 837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579 (at 10.9.104.22@o2ib4) reconnecting [Thu Apr 11 08:18:04 2019][2840804.785928] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 08:22:22 2019][2841062.238835] Lustre: 117550:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0001: Failure to clear the changelog for user 1: -22 [Thu Apr 11 08:22:22 2019][2841062.250769] Lustre: 117550:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 30 previous similar messages [Thu Apr 11 08:27:48 2019][2841388.837052] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Thu Apr 11 08:27:48 2019][2841388.844378] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 08:28:04 2019][2841404.788615] Lustre: fir-MDT0002: Client 58899832-571e-4a18-664b-6f5c1308e27e (at 10.8.18.18@o2ib6) reconnecting [Thu Apr 11 08:28:04 2019][2841404.798874] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 08:36:02 2019][2841882.455049] LustreError: 118026:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b609732e600 x1628639256416336/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:8/0 lens 448/440 e 0 to 0 dl 1554996968 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 08:37:48 2019][2841988.862554] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.24@o2ib6) [Thu Apr 11 08:37:48 2019][2841988.869875] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 08:38:04 2019][2842004.834110] Lustre: fir-MDT0002: Client 67753486-aa02-1106-6851-77990d7e25c7 (at 10.9.107.61@o2ib4) reconnecting [Thu Apr 11 08:38:04 2019][2842004.844466] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 08:39:53 2019][2842113.467381] LustreError: 119190:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b5f8874cb00 x1628639256836656/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:29/0 lens 448/440 e 0 to 0 dl 1554997199 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 08:47:33 2019][2842573.507403] Lustre: 118243:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 08:47:49 2019][2842589.829904] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.60@o2ib4) [Thu Apr 11 08:47:49 2019][2842589.837395] Lustre: Skipped 1534 previous similar messages [Thu Apr 11 08:48:07 2019][2842607.032558] Lustre: fir-MDT0002: Client 6d84db07-946f-3a55-fea2-55904c526e1b (at 10.8.1.21@o2ib6) reconnecting [Thu Apr 11 08:48:07 2019][2842607.042735] Lustre: Skipped 1533 previous similar messages [Thu Apr 11 08:57:44 2019][2843184.663840] LustreError: 119096:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b79473d6600 x1628639258765312/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:20/0 lens 448/440 e 0 to 0 dl 1554998270 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 08:57:50 2019][2843190.190960] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 08:57:50 2019][2843190.201488] Lustre: Skipped 1538 previous similar messages [Thu Apr 11 08:58:07 2019][2843207.429282] Lustre: fir-MDT0002: Client 3c1359c4-6f8a-28e8-3fe1-a0a63b3ad896 (at 10.9.102.1@o2ib4) reconnecting [Thu Apr 11 08:58:07 2019][2843207.439544] Lustre: Skipped 1534 previous similar messages [Thu Apr 11 08:58:26 2019][2843226.680540] LustreError: 119103:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b75266c2c50 x1628639258841616/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:2/0 lens 448/440 e 0 to 0 dl 1554998312 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 09:07:50 2019][2843790.620010] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.22@o2ib6) [Thu Apr 11 09:07:50 2019][2843790.627330] Lustre: Skipped 1526 previous similar messages [Thu Apr 11 09:08:08 2019][2843808.086686] Lustre: fir-MDT0002: Client e18a93c6-a11b-0b52-af65-1483de6beb82 (at 10.8.8.18@o2ib6) reconnecting [Thu Apr 11 09:08:08 2019][2843808.096870] Lustre: Skipped 1530 previous similar messages [Thu Apr 11 09:17:11 2019][2844351.019110] Lustre: fir-MDT0000: haven't heard from client 062b6f4e-85d8-c11f-5a12-27ec1dc7a3ae (at 10.9.107.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b56cabb5c00, cur 1554999431 expire 1554999281 last 1554999204 [Thu Apr 11 09:17:11 2019][2844351.041079] Lustre: Skipped 3 previous similar messages [Thu Apr 11 09:17:15 2019][2844355.019096] Lustre: fir-MDT0002: haven't heard from client 062b6f4e-85d8-c11f-5a12-27ec1dc7a3ae (at 10.9.107.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8aee741400, cur 1554999435 expire 1554999285 last 1554999208 [Thu Apr 11 09:17:15 2019][2844355.041062] Lustre: Skipped 3 previous similar messages [Thu Apr 11 09:17:52 2019][2844392.315266] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Thu Apr 11 09:17:52 2019][2844392.322587] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 09:18:08 2019][2844408.196526] Lustre: fir-MDT0002: Client 8edb63b6-d814-54d5-ddee-ddb62f3308b6 (at 10.9.103.42@o2ib4) reconnecting [Thu Apr 11 09:18:08 2019][2844408.206879] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 09:27:52 2019][2844992.753337] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.1@o2ib4) [Thu Apr 11 09:27:52 2019][2844992.760753] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 09:28:08 2019][2845008.707356] Lustre: fir-MDT0002: Client e4a8aa81-b0b9-5652-c269-3fdf6ff70c2f (at 10.9.102.23@o2ib4) reconnecting [Thu Apr 11 09:28:08 2019][2845008.717711] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 09:37:53 2019][2845593.607663] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.22@o2ib4) [Thu Apr 11 09:37:53 2019][2845593.615163] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 09:38:08 2019][2845608.819182] Lustre: fir-MDT0002: Client ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4) reconnecting [Thu Apr 11 09:38:08 2019][2845608.829529] Lustre: Skipped 1534 previous similar messages [Thu Apr 11 09:47:53 2019][2846193.837407] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 09:47:53 2019][2846193.847933] Lustre: Skipped 1543 previous similar messages [Thu Apr 11 09:48:08 2019][2846209.012112] Lustre: fir-MDT0002: Client a11a9980-d36a-715f-c190-a6765cde5927 (at 10.8.1.22@o2ib6) reconnecting [Thu Apr 11 09:48:08 2019][2846209.022288] Lustre: Skipped 1526 previous similar messages [Thu Apr 11 09:57:54 2019][2846794.375581] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Thu Apr 11 09:57:54 2019][2846794.385964] Lustre: Skipped 1523 previous similar messages [Thu Apr 11 09:58:09 2019][2846809.622214] Lustre: fir-MDT0002: Client 1fed4b9d-ca5d-165d-887c-2f1234134d0c (at 10.8.9.10@o2ib6) reconnecting [Thu Apr 11 09:58:09 2019][2846809.632393] Lustre: Skipped 1534 previous similar messages [Thu Apr 11 10:07:55 2019][2847395.788203] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Thu Apr 11 10:07:55 2019][2847395.795520] Lustre: Skipped 1539 previous similar messages [Thu Apr 11 10:08:09 2019][2847409.690865] Lustre: fir-MDT0002: Client 10601291-a183-8deb-41a7-af6620223334 (at 10.9.108.36@o2ib4) reconnecting [Thu Apr 11 10:08:09 2019][2847409.701224] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 10:16:09 2019][2847889.302552] LustreError: 119197:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6c2fcaa850 x1628639267285664/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:15/0 lens 448/440 e 0 to 0 dl 1555002975 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 10:16:10 2019][2847890.646239] Lustre: 118363:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 10:16:10 2019][2847890.658152] Lustre: 118363:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 4 previous similar messages [Thu Apr 11 10:17:56 2019][2847996.179937] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.41@o2ib4) [Thu Apr 11 10:17:56 2019][2847996.187430] Lustre: Skipped 1532 previous similar messages [Thu Apr 11 10:18:09 2019][2848010.042634] Lustre: fir-MDT0002: Client eda11444-0027-aea4-d226-92aced58871b (at 10.9.114.5@o2ib4) reconnecting [Thu Apr 11 10:18:09 2019][2848010.052915] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 10:18:57 2019][2848057.372390] LustreError: 118512:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8b15a42a00 x1628639267592720/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:3/0 lens 448/440 e 0 to 0 dl 1555003143 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 10:18:58 2019][2848058.644906] Lustre: 118299:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 10:18:58 2019][2848058.656820] Lustre: 118299:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Apr 11 10:24:48 2019][2848408.104856] Lustre: fir-MDT0000: haven't heard from client 83b48ed7-6f05-aa34-fa68-b4aa8620446e (at 10.9.115.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4d085d9800, cur 1555003488 expire 1555003338 last 1555003261 [Thu Apr 11 10:27:00 2019][2848540.455716] LustreError: 119189:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b7cb6336050 x1628639268477152/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:6/0 lens 448/440 e 0 to 0 dl 1555003626 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 10:27:52 2019][2848592.107775] Lustre: fir-MDT0000: haven't heard from client 258f183a-0fad-0e93-2172-803123bfd4b3 (at 10.8.19.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4f4eb2cc00, cur 1555003672 expire 1555003522 last 1555003445 [Thu Apr 11 10:27:52 2019][2848592.129573] Lustre: Skipped 4 previous similar messages [Thu Apr 11 10:27:56 2019][2848596.425940] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.22@o2ib6) [Thu Apr 11 10:27:56 2019][2848596.433257] Lustre: Skipped 1528 previous similar messages [Thu Apr 11 10:28:10 2019][2848610.092764] Lustre: fir-MDT0002: Client f53d399b-a6b5-36d0-034f-8c8c0efbb65f (at 10.9.106.11@o2ib4) reconnecting [Thu Apr 11 10:28:10 2019][2848610.103136] Lustre: Skipped 1536 previous similar messages [Thu Apr 11 10:35:19 2019][2849039.117069] Lustre: fir-MDT0000: haven't heard from client 6ae4bb58-5b57-83f9-f55c-4525713a6584 (at 10.8.26.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4f4eb29000, cur 1555004119 expire 1555003969 last 1555003892 [Thu Apr 11 10:35:19 2019][2849039.138944] Lustre: Skipped 4 previous similar messages [Thu Apr 11 10:35:29 2019][2849049.117642] Lustre: fir-MDT0002: haven't heard from client 6ae4bb58-5b57-83f9-f55c-4525713a6584 (at 10.8.26.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b87dd6d1c00, cur 1555004129 expire 1555003979 last 1555003902 [Thu Apr 11 10:35:29 2019][2849049.139522] Lustre: Skipped 3 previous similar messages [Thu Apr 11 10:37:56 2019][2849196.666125] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Thu Apr 11 10:37:56 2019][2849196.676574] Lustre: Skipped 1539 previous similar messages [Thu Apr 11 10:38:02 2019][2849202.138536] Lustre: 118963:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628641945535088 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b6ce58f9450 x1628641945535088/t0(0) o4->83b8ff7e-e264-e3fb-e7eb-42bd933886ea@10.8.1.23@o2ib6:8/0 lens 640/0 e 0 to 0 dl 1555004288 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 10:38:02 2019][2849202.167958] Lustre: 118963:0:(tgt_handler.c:562:tgt_handle_recovery()) Skipped 2 previous similar messages [Thu Apr 11 10:38:10 2019][2849210.484728] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Thu Apr 11 10:38:10 2019][2849210.495187] Lustre: Skipped 1525 previous similar messages [Thu Apr 11 10:40:57 2019][2849377.124187] Lustre: fir-MDT0000: haven't heard from client 404c2558-f8c5-5468-4f54-4c6b18f8651a (at 10.8.23.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4cc9c69400, cur 1555004457 expire 1555004307 last 1555004230 [Thu Apr 11 10:45:17 2019][2849637.342378] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 10:45:33 2019][2849653.556169] LustreError: 119179:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6278eaaa00 x1628639271055824/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:9/0 lens 448/440 e 0 to 0 dl 1555004739 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 10:47:57 2019][2849797.809628] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Thu Apr 11 10:47:57 2019][2849797.819971] Lustre: Skipped 1527 previous similar messages [Thu Apr 11 10:48:10 2019][2849810.880781] Lustre: fir-MDT0002: Client 94d4bb8d-4025-4bb2-3e79-daa06ff1054f (at 10.9.108.15@o2ib4) reconnecting [Thu Apr 11 10:48:10 2019][2849810.891143] Lustre: Skipped 1535 previous similar messages [Thu Apr 11 10:50:06 2019][2849926.642052] LustreError: 119081:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b72afa09200 x1628639271630336/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:12/0 lens 448/440 e 0 to 0 dl 1555005012 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 10:57:59 2019][2850399.233913] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.21@o2ib6) [Thu Apr 11 10:57:59 2019][2850399.241228] Lustre: Skipped 1545 previous similar messages [Thu Apr 11 10:58:10 2019][2850411.103511] Lustre: fir-MDT0002: Client 91934493-be40-5b23-a558-2943654113ab (at 10.8.27.16@o2ib6) reconnecting [Thu Apr 11 10:58:11 2019][2850411.113825] Lustre: Skipped 1529 previous similar messages [Thu Apr 11 11:06:12 2019][2850892.835250] LustreError: 119269:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012114:0x222:0x0] [Thu Apr 11 11:06:19 2019][2850899.870997] LustreError: 118602:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00106ba:0x1653:0x0] [Thu Apr 11 11:06:27 2019][2850907.576419] LustreError: 118951:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b72c66c8450 x1628648783554816/t0(0) o4->837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579@10.9.104.22@o2ib4:3/0 lens 488/448 e 0 to 0 dl 1555005993 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:06:27 2019][2850907.576787] Lustre: fir-MDT0002: Bulk IO write error with 837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579 (at 10.9.104.22@o2ib4), client will retry: rc = -107 [Thu Apr 11 11:06:27 2019][2850907.614273] LustreError: 118951:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 1 previous similar message [Thu Apr 11 11:07:59 2019][2850999.316770] Lustre: fir-MDT0002: Connection restored to 15562cac-eb22-f844-7b88-546ba9ddee38 (at 10.9.101.71@o2ib4) [Thu Apr 11 11:07:59 2019][2850999.327390] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 11:08:11 2019][2851011.154555] Lustre: fir-MDT0002: Client a132911f-e28d-2e04-1263-47066608ec3c (at 10.8.23.24@o2ib6) reconnecting [Thu Apr 11 11:08:11 2019][2851011.164827] Lustre: Skipped 1526 previous similar messages [Thu Apr 11 11:08:55 2019][2851056.126345] Lustre: 118227:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 11:08:56 2019][2851056.138296] Lustre: 118227:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 10 previous similar messages [Thu Apr 11 11:09:56 2019][2851116.205222] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 11:10:05 2019][2851125.989852] LustreError: 118657:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b7077245450 x1630280096470480/t0(0) o4->6ea7ad88-4c41-3a56-d580-e39b0e85132b@10.8.27.6@o2ib6:11/0 lens 488/448 e 0 to 0 dl 1555006211 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:10:05 2019][2851126.014538] Lustre: fir-MDT0002: Bulk IO write error with 6ea7ad88-4c41-3a56-d580-e39b0e85132b (at 10.8.27.6@o2ib6), client will retry: rc = -107 [Thu Apr 11 11:10:05 2019][2851126.027833] Lustre: Skipped 1 previous similar message [Thu Apr 11 11:11:17 2019][2851197.743095] Lustre: 118549:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 11:11:40 2019][2851220.577685] Lustre: 118414:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 11:11:40 2019][2851220.589602] Lustre: 118414:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 7 previous similar messages [Thu Apr 11 11:15:16 2019][2851436.241190] Lustre: 118318:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 11:16:13 2019][2851493.794939] Lustre: 118361:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 11:16:13 2019][2851493.806868] Lustre: 118361:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 17 previous similar messages [Thu Apr 11 11:16:22 2019][2851502.386820] Lustre: 118369:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 11:16:22 2019][2851502.398765] Lustre: 118369:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 9 previous similar messages [Thu Apr 11 11:17:52 2019][2851592.580096] Lustre: 118235:0:(mdd_device.c:1794:mdd_changelog_clear()) fir-MDD0002: Failure to clear the changelog for user 1: -22 [Thu Apr 11 11:17:52 2019][2851592.592049] Lustre: 118235:0:(mdd_device.c:1794:mdd_changelog_clear()) Skipped 1 previous similar message [Thu Apr 11 11:17:59 2019][2851600.105650] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.60@o2ib4) [Thu Apr 11 11:17:59 2019][2851600.113140] Lustre: Skipped 1531 previous similar messages [Thu Apr 11 11:18:11 2019][2851611.347518] Lustre: fir-MDT0002: Client bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83 (at 10.8.26.21@o2ib6) reconnecting [Thu Apr 11 11:18:11 2019][2851611.357799] Lustre: Skipped 1541 previous similar messages [Thu Apr 11 11:20:42 2019][2851762.231119] LustreError: 118747:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001116b:0xa17:0x0] [Thu Apr 11 11:21:16 2019][2851796.487169] LustreError: 118793:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010863:0x898:0x0] [Thu Apr 11 11:24:33 2019][2851993.459429] Lustre: 117491:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 11:24:33 2019][2851993.459429] req@ffff8b5b6d35f200 x1628642479172592/t146276271496(0) o36->6d84db07-946f-3a55-fea2-55904c526e1b@10.8.1.21@o2ib6:8/0 lens 536/2888 e 1 to 0 dl 1555007078 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:24:33 2019][2851993.489326] Lustre: 117491:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Thu Apr 11 11:25:48 2019][2852068.429929] LustreError: 118177:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555007058, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b54ccb96300/0x53d0f21983285b0b lrc: 3/1,0 mode: --/PR res: [0x2c0010636:0x604a:0x0].0x0 bits 0x13/0x8 rrc: 11 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118177 timeout: 0 lvb_type: 0 [Thu Apr 11 11:25:48 2019][2852068.469788] LustreError: 118177:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 11:27:04 2019][2852144.195365] LustreError: 119078:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107a5:0x5650:0x0] [Thu Apr 11 11:27:38 2019][2852178.733159] LNet: Service thread pid 118308 was inactive for 200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 11:27:38 2019][2852178.750385] LNet: Skipped 1 previous similar message [Thu Apr 11 11:27:38 2019][2852178.755542] Pid: 118308, comm: mdt00_058 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 11:27:38 2019][2852178.765553] Call Trace: [Thu Apr 11 11:27:38 2019][2852178.768194] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 11:27:38 2019][2852178.775333] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 11:27:38 2019][2852178.782711] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 11:27:38 2019][2852178.789644] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 11:27:38 2019][2852178.796310] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 11:27:38 2019][2852178.802552] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 11:27:38 2019][2852178.809301] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 11:27:38 2019][2852178.815175] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 11:27:38 2019][2852178.822309] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 11:27:38 2019][2852178.830204] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 11:27:38 2019][2852178.836716] [] kthread+0xd1/0xe0 [Thu Apr 11 11:27:38 2019][2852178.841810] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 11:27:38 2019][2852178.848475] [] 0xffffffffffffffff [Thu Apr 11 11:27:38 2019][2852178.853670] LustreError: dumping log to /tmp/lustre-log.1555007258.118308 [Thu Apr 11 11:27:40 2019][2852180.382095] Pid: 118177, comm: mdt00_017 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 11:27:40 2019][2852180.392115] Call Trace: [Thu Apr 11 11:27:40 2019][2852180.394770] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 11:27:40 2019][2852180.401967] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 11:27:40 2019][2852180.409412] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 11:27:40 2019][2852180.416419] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 11:27:40 2019][2852180.423601] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 11:27:40 2019][2852180.430254] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 11:27:40 2019][2852180.437424] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 11:27:40 2019][2852180.444169] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 11:27:40 2019][2852180.450831] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 11:27:40 2019][2852180.457781] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 11:27:40 2019][2852180.465070] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 11:27:40 2019][2852180.471433] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 11:27:40 2019][2852180.478548] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 11:27:40 2019][2852180.486451] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 11:27:40 2019][2852180.492962] [] kthread+0xd1/0xe0 [Thu Apr 11 11:27:40 2019][2852180.498057] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 11:27:40 2019][2852180.504705] [] 0xffffffffffffffff [Thu Apr 11 11:28:01 2019][2852201.148324] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 11:28:01 2019][2852201.158849] Lustre: Skipped 1552 previous similar messages [Thu Apr 11 11:28:11 2019][2852211.373700] Lustre: fir-MDT0002: Client 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4) reconnecting [Thu Apr 11 11:28:11 2019][2852211.384056] Lustre: Skipped 1537 previous similar messages [Thu Apr 11 11:35:32 2019][2852652.433107] Lustre: 118928:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628640699697392 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b551cef3450 x1628640699697392/t0(0) o4->398eea62-82fb-c936-bc1a-b5cbed90bbf5@10.8.8.12@o2ib6:8/0 lens 1136/0 e 0 to 0 dl 1555007738 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 11:35:51 2019][2852671.275871] LustreError: 119037:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b52af30e300 x1629298046237264/t0(0) o37->f5140abb-0b33-a9a1-9459-14bf253ba61e@10.9.106.60@o2ib4:27/0 lens 448/440 e 0 to 0 dl 1555007757 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:38:01 2019][2852801.203243] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Thu Apr 11 11:38:01 2019][2852801.213618] Lustre: Skipped 1556 previous similar messages [Thu Apr 11 11:38:11 2019][2852811.679707] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Thu Apr 11 11:38:11 2019][2852811.689979] Lustre: Skipped 1562 previous similar messages [Thu Apr 11 11:39:16 2019][2852876.194165] Lustre: fir-MDT0002: haven't heard from client 83661770-f8f4-e73b-ca15-a24428bd76a4 (at 10.9.115.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b79381b6000, cur 1555007956 expire 1555007806 last 1555007729 [Thu Apr 11 11:39:16 2019][2852876.216042] Lustre: Skipped 4 previous similar messages [Thu Apr 11 11:47:44 2019][2853385.122976] LustreError: 118904:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8ccd78f450 x1629293174282048/t0(0) o3->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:14/0 lens 488/440 e 0 to 0 dl 1555008494 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:47:44 2019][2853385.147530] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 11:48:01 2019][2853401.356768] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Thu Apr 11 11:48:01 2019][2853401.364150] Lustre: Skipped 1561 previous similar messages [Thu Apr 11 11:48:12 2019][2853412.930160] Lustre: fir-MDT0002: Client 86cc2579-6592-d6be-ad47-10ff366dfbdf (at 10.8.8.19@o2ib6) reconnecting [Thu Apr 11 11:48:12 2019][2853412.940350] Lustre: Skipped 1561 previous similar messages [Thu Apr 11 11:51:22 2019][2853602.297175] Lustre: 118630:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629293176108832 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b5a5a914050 x1629293176108832/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:22/0 lens 880/0 e 0 to 0 dl 1555008712 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 11:51:22 2019][2853602.307887] LustreError: 118655:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b7211f40450 x1629293176108800/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:22/0 lens 488/448 e 0 to 0 dl 1555008712 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:51:22 2019][2853602.307903] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 11:53:49 2019][2853749.195847] LustreError: 118542:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8ccee7a400 x1628639282225808/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:24/0 lens 448/440 e 0 to 0 dl 1555008834 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:56:39 2019][2853919.847801] Lustre: 118317:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 11:56:39 2019][2853919.847801] req@ffff8b64b672b600 x1630376526906240/t0(0) o101->58eee3b6-9427-4137-1283-a0bee5ae7fd4@10.9.106.5@o2ib4:14/0 lens 632/3264 e 1 to 0 dl 1555009004 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:56:39 2019][2853919.876969] Lustre: 118317:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 11:56:49 2019][2853929.868005] Lustre: 118389:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 11:56:49 2019][2853929.868005] req@ffff8b69ba757200 x1630272990343168/t146289134890(0) o36->aa1e9ce5-2ac3-cfae-1629-e9796f53cb3b@10.9.107.30@o2ib4:24/0 lens 544/2888 e 0 to 0 dl 1555009014 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:56:57 2019][2853937.442159] Lustre: 118441:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 11:56:57 2019][2853937.442159] req@ffff8b58046eec00 x1629253126332320/t0(0) o101->bafe047a-89ea-21d5-a24e-5c407648cfc7@10.0.10.3@o2ib7:2/0 lens 576/3264 e 0 to 0 dl 1555009022 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:57:54 2019][2853994.961305] LustreError: 118333:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555008984, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b6c597bde80/0x53d0f219b3be72c2 lrc: 3/1,0 mode: --/PR res: [0x2c001289c:0x12f6:0x0].0x0 bits 0x13/0x8 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118333 timeout: 0 lvb_type: 0 [Thu Apr 11 11:57:54 2019][2853995.001257] LustreError: 118333:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 11:58:02 2019][2854002.427280] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.1@o2ib4) [Thu Apr 11 11:58:02 2019][2854002.434699] Lustre: Skipped 1572 previous similar messages [Thu Apr 11 11:58:02 2019][2854002.438453] LustreError: 118469:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555008992, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0003_UUID lock: ffff8b596ea6ca40/0x53d0f219b3e96e3d lrc: 3/1,0 mode: --/PR res: [0x280010253:0xe3c:0x0].0x0 bits 0x12/0x0 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118469 timeout: 0 lvb_type: 0 [Thu Apr 11 11:58:13 2019][2854013.671495] Lustre: fir-MDT0002: Client 1fed4b9d-ca5d-165d-887c-2f1234134d0c (at 10.8.9.10@o2ib6) reconnecting [Thu Apr 11 11:58:13 2019][2854013.681671] Lustre: Skipped 1573 previous similar messages [Thu Apr 11 11:59:07 2019][2854067.354852] LustreError: 119140:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6c32bee900 x1629293178920736/t0(0) o37->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:13/0 lens 448/440 e 0 to 0 dl 1555009153 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 11:59:45 2019][2854105.427483] LNet: Service thread pid 118189 was inactive for 200.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 11:59:45 2019][2854105.444695] LNet: Skipped 1 previous similar message [Thu Apr 11 11:59:45 2019][2854105.449843] Pid: 118189, comm: mdt01_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 11:59:45 2019][2854105.459852] Call Trace: [Thu Apr 11 11:59:45 2019][2854105.462493] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 11:59:45 2019][2854105.469617] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 11:59:45 2019][2854105.476994] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 11:59:45 2019][2854105.483918] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 11:59:45 2019][2854105.490572] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 11:59:45 2019][2854105.496795] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 11:59:45 2019][2854105.503538] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 11:59:45 2019][2854105.509414] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 11:59:45 2019][2854105.516576] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 11:59:45 2019][2854105.524489] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 11:59:45 2019][2854105.530996] [] kthread+0xd1/0xe0 [Thu Apr 11 11:59:45 2019][2854105.536082] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 11:59:45 2019][2854105.542736] [] 0xffffffffffffffff [Thu Apr 11 11:59:45 2019][2854105.547933] LustreError: dumping log to /tmp/lustre-log.1555009185.118189 [Thu Apr 11 11:59:46 2019][2854106.942101] LNet: Service thread pid 118333 was inactive for 201.97s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 11:59:46 2019][2854106.959327] Pid: 118333, comm: mdt01_071 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 11:59:46 2019][2854106.969328] Call Trace: [Thu Apr 11 11:59:46 2019][2854106.971965] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 11:59:46 2019][2854106.979080] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 11:59:46 2019][2854106.986457] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 11:59:46 2019][2854106.993474] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 11:59:46 2019][2854107.000652] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 11:59:46 2019][2854107.007315] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 11:59:46 2019][2854107.014494] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 11:59:46 2019][2854107.021237] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 11:59:46 2019][2854107.027972] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 11:59:46 2019][2854107.034932] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 11:59:46 2019][2854107.042219] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 11:59:46 2019][2854107.048563] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 11:59:46 2019][2854107.055679] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 11:59:46 2019][2854107.063565] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 11:59:46 2019][2854107.070092] [] kthread+0xd1/0xe0 [Thu Apr 11 11:59:46 2019][2854107.075179] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 11:59:46 2019][2854107.081826] [] 0xffffffffffffffff [Thu Apr 11 11:59:52 2019][2854112.595644] LNet: Service thread pid 118469 was inactive for 200.15s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 11:59:52 2019][2854112.612847] Pid: 118469, comm: mdt00_087 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 11:59:52 2019][2854112.622854] Call Trace: [Thu Apr 11 11:59:52 2019][2854112.625492] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.632597] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.639971] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 11:59:52 2019][2854112.646985] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 11:59:52 2019][2854112.654159] [] mdt_getattr_name_lock+0x11d/0x1c30 [mdt] [Thu Apr 11 11:59:52 2019][2854112.661258] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 11:59:52 2019][2854112.668000] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 11:59:52 2019][2854112.674657] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.681610] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.688916] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.695260] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.702374] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.710287] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.716835] [] kthread+0xd1/0xe0 [Thu Apr 11 11:59:52 2019][2854112.721929] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 11:59:52 2019][2854112.728584] [] 0xffffffffffffffff [Thu Apr 11 11:59:52 2019][2854112.733777] LustreError: dumping log to /tmp/lustre-log.1555009192.118469 [Thu Apr 11 11:59:52 2019][2854112.764426] Pid: 118205, comm: mdt01_026 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 11:59:52 2019][2854112.774430] Call Trace: [Thu Apr 11 11:59:52 2019][2854112.777073] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.784193] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.791577] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 11:59:52 2019][2854112.798509] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 11:59:52 2019][2854112.805165] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 11:59:52 2019][2854112.811389] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 11:59:52 2019][2854112.818151] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 11:59:52 2019][2854112.824032] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.831162] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.839059] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 11:59:52 2019][2854112.845559] [] kthread+0xd1/0xe0 [Thu Apr 11 11:59:52 2019][2854112.850647] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 11:59:52 2019][2854112.857293] [] 0xffffffffffffffff [Thu Apr 11 12:00:11 2019][2854132.031257] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 12:01:11 2019][2854191.365159] LustreError: 119004:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5a5a913450 x1629293179640080/t0(0) o3->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:11/0 lens 488/440 e 0 to 0 dl 1555009301 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:01:11 2019][2854191.389727] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 12:01:42 2019][2854222.349319] LustreError: 118028:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b70897e2a00 x1629293179808368/t0(0) o37->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:18/0 lens 448/440 e 0 to 0 dl 1555009308 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:05:19 2019][2854439.376749] LustreError: 118888:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8ccd78b050 x1629293180872688/t0(0) o3->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:24/0 lens 488/440 e 0 to 0 dl 1555009524 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:05:19 2019][2854439.401247] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 12:05:58 2019][2854478.195455] LustreError: 119055:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b6c68679450 x1628648788630768/t0(0) o4->837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579@10.9.104.22@o2ib4:27/0 lens 488/448 e 0 to 0 dl 1555009587 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:05:58 2019][2854478.206321] Lustre: 119021:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628648788630784 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b5984465050 x1628648788630784/t0(0) o4->837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579@10.9.104.22@o2ib4:27/0 lens 496/0 e 0 to 0 dl 1555009587 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 12:05:58 2019][2854478.249631] Lustre: fir-MDT0002: Bulk IO write error with 837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579 (at 10.9.104.22@o2ib4), client will retry: rc = -110 [Thu Apr 11 12:08:02 2019][2854602.759379] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Thu Apr 11 12:08:02 2019][2854602.766798] Lustre: Skipped 1646 previous similar messages [Thu Apr 11 12:08:13 2019][2854614.086371] Lustre: fir-MDT0002: Client 10601291-a183-8deb-41a7-af6620223334 (at 10.9.108.36@o2ib4) reconnecting [Thu Apr 11 12:08:13 2019][2854614.096742] Lustre: Skipped 1648 previous similar messages [Thu Apr 11 12:09:13 2019][2854673.368142] LustreError: 119190:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b66cedd9b00 x1628639285262704/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:19/0 lens 448/440 e 0 to 0 dl 1555009759 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:12:02 2019][2854842.513947] LustreError: 117476:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b536ca4f200 x1629293182524544/t0(0) o37->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:2/0 lens 448/440 e 0 to 0 dl 1555009952 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:16:05 2019][2855085.237674] Lustre: fir-MDT0000: haven't heard from client 98b7d18a-8861-5f82-983d-c060d82a9e0e (at 10.9.108.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b677fa4c800, cur 1555010165 expire 1555010015 last 1555009938 [Thu Apr 11 12:16:05 2019][2855085.259650] Lustre: Skipped 4 previous similar messages [Thu Apr 11 12:16:41 2019][2855121.600129] LustreError: 118747:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5a5a910450 x1629293183552224/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:11/0 lens 488/448 e 0 to 0 dl 1555010231 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:16:41 2019][2855121.624762] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 12:17:21 2019][2855161.240082] Lustre: fir-MDT0000: haven't heard from client f0b5d5e5-57ad-9ea0-b7d1-cc88bf8b90d7 (at 10.9.108.54@o2ib4) in 212 seconds. I think it's dead, and I am evicting it. exp ffff8b4e06f47800, cur 1555010241 expire 1555010091 last 1555010029 [Thu Apr 11 12:17:21 2019][2855161.262046] Lustre: Skipped 4 previous similar messages [Thu Apr 11 12:17:36 2019][2855176.240042] Lustre: fir-MDT0001: haven't heard from client f0b5d5e5-57ad-9ea0-b7d1-cc88bf8b90d7 (at 10.9.108.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8cef352c00, cur 1555010256 expire 1555010106 last 1555010029 [Thu Apr 11 12:17:36 2019][2855176.262006] Lustre: Skipped 47 previous similar messages [Thu Apr 11 12:18:02 2019][2855202.955719] Lustre: fir-MDT0002: Connection restored to 3dde01db-8bab-e6a5-6af0-a17c56d94df2 (at 10.9.101.66@o2ib4) [Thu Apr 11 12:18:02 2019][2855202.966349] Lustre: Skipped 1655 previous similar messages [Thu Apr 11 12:18:14 2019][2855214.249490] Lustre: fir-MDT0002: Client eda11444-0027-aea4-d226-92aced58871b (at 10.9.114.5@o2ib4) reconnecting [Thu Apr 11 12:18:14 2019][2855214.259779] Lustre: Skipped 1644 previous similar messages [Thu Apr 11 12:18:37 2019][2855237.240570] Lustre: fir-MDT0000: haven't heard from client 4f6cf8db-defd-fe4d-13a5-13a531f83d9e (at 10.9.106.51@o2ib4) in 225 seconds. I think it's dead, and I am evicting it. exp ffff8b4cc3930c00, cur 1555010317 expire 1555010167 last 1555010092 [Thu Apr 11 12:18:37 2019][2855237.262547] Lustre: Skipped 35 previous similar messages [Thu Apr 11 12:18:52 2019][2855252.240883] Lustre: fir-MDT0001: haven't heard from client 3187b0e1-ff81-6f7b-0956-81660a0ee3a3 (at 10.9.112.11@o2ib4) in 185 seconds. I think it's dead, and I am evicting it. exp ffff8b5c066a3000, cur 1555010332 expire 1555010182 last 1555010147 [Thu Apr 11 12:18:52 2019][2855252.263212] Lustre: Skipped 8 previous similar messages [Thu Apr 11 12:20:35 2019][2855355.848426] LustreError: 118958:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012966:0x1c9b:0x0] [Thu Apr 11 12:21:28 2019][2855408.467722] LustreError: 119151:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b82b8381500 x1628639287502048/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:28/0 lens 448/440 e 0 to 0 dl 1555010518 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:21:28 2019][2855408.492294] LustreError: 119151:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 3 previous similar messages [Thu Apr 11 12:21:51 2019][2855431.621909] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 12:22:53 2019][2855493.621126] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 12:23:45 2019][2855545.246718] Lustre: fir-MDT0000: haven't heard from client e67d9408-d6aa-f2b1-6ad6-99df52a43f79 (at 10.9.115.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8cfeee2400, cur 1555010625 expire 1555010475 last 1555010398 [Thu Apr 11 12:23:45 2019][2855545.268608] Lustre: Skipped 1 previous similar message [Thu Apr 11 12:24:26 2019][2855586.607252] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 12:28:02 2019][2855802.987288] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.12@o2ib6) [Thu Apr 11 12:28:02 2019][2855802.994912] Lustre: Skipped 1647 previous similar messages [Thu Apr 11 12:28:14 2019][2855814.435014] Lustre: fir-MDT0002: Client f53d399b-a6b5-36d0-034f-8c8c0efbb65f (at 10.9.106.11@o2ib4) reconnecting [Thu Apr 11 12:28:14 2019][2855814.445380] Lustre: Skipped 1651 previous similar messages [Thu Apr 11 12:30:07 2019][2855927.758091] Lustre: 118617:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629293186091024 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b52e0668050 x1629293186091024/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:7/0 lens 872/0 e 0 to 0 dl 1555011037 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 12:31:40 2019][2856020.698551] LustreError: 119223:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b69e7224c50 x1629293186363904/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:10/0 lens 488/448 e 0 to 0 dl 1555011130 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:31:40 2019][2856020.723120] LustreError: 119223:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 6 previous similar messages [Thu Apr 11 12:31:40 2019][2856020.732795] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 12:34:30 2019][2856190.700680] LustreError: 118754:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b6a2b3f8450 x1629297425981584/t0(0) o4->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:0/0 lens 488/448 e 0 to 0 dl 1555011300 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:34:30 2019][2856190.700801] Lustre: fir-MDT0002: Bulk IO write error with a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6), client will retry: rc = -110 [Thu Apr 11 12:34:30 2019][2856190.737946] LustreError: 118754:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message [Thu Apr 11 12:35:48 2019][2856268.775458] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 12:36:31 2019][2856311.575370] LustreError: 119242:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b7b0d2dfc50 x1628639290771520/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:7/0 lens 448/440 e 0 to 0 dl 1555011397 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:36:50 2019][2856330.687573] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 12:38:02 2019][2856403.059946] Lustre: fir-MDT0002: Connection restored to 6e57a6e6-07ae-4b36-8d72-608075772c63 (at 10.9.106.55@o2ib4) [Thu Apr 11 12:38:02 2019][2856403.071294] Lustre: Skipped 1648 previous similar messages [Thu Apr 11 12:38:14 2019][2856414.456038] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Thu Apr 11 12:38:14 2019][2856414.466414] Lustre: Skipped 1646 previous similar messages [Thu Apr 11 12:40:58 2019][2856578.918008] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 12:40:58 2019][2856578.931528] Lustre: Skipped 1 previous similar message [Thu Apr 11 12:41:06 2019][2856587.108615] Lustre: 117465:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 12:41:06 2019][2856587.108615] req@ffff8b557a3d6900 x1628640526944032/t146303166158(0) o36->f87bfb7b-5313-2d83-7524-cc6a927681f0@10.9.108.65@o2ib4:11/0 lens 544/2888 e 1 to 0 dl 1555011671 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:41:06 2019][2856587.138873] Lustre: 117465:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 12:41:16 2019][2856596.338804] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 12:41:16 2019][2856596.338804] req@ffff8b6743a5f200 x1628548351688704/t0(0) o101->abc2f529-d1a0-9c54-fdd0-ac08914afbb0@10.8.7.13@o2ib6:21/0 lens 576/3264 e 0 to 0 dl 1555011681 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:41:37 2019][2856617.720625] LNetError: 117119:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 12:41:40 2019][2856620.268286] Lustre: fir-MDT0000: haven't heard from client dffafbfb-688f-8a2a-1763-61bf9f7e639e (at 10.9.101.44@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5c7322f800, cur 1555011700 expire 1555011550 last 1555011473 [Thu Apr 11 12:41:40 2019][2856620.290251] Lustre: Skipped 4 previous similar messages [Thu Apr 11 12:41:50 2019][2856630.709416] LustreError: 118498:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b88e6263f00 x1628639292078144/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:20/0 lens 448/440 e 0 to 0 dl 1555011740 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:41:50 2019][2856630.733994] LustreError: 118498:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 7 previous similar messages [Thu Apr 11 12:42:22 2019][2856662.267139] LustreError: 117467:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555011651, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b8cc4e75e80/0x53d0f219ed448a89 lrc: 3/1,0 mode: --/PR res: [0x2c0010830:0x1107c:0x0].0x0 bits 0x13/0x8 rrc: 12 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 117467 timeout: 0 lvb_type: 0 [Thu Apr 11 12:42:22 2019][2856662.307131] LustreError: 117467:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Thu Apr 11 12:42:31 2019][2856671.897906] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 12:42:58 2019][2856698.948420] LustreError: 117517:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011227:0x17236:0x0] [Thu Apr 11 12:43:01 2019][2856701.791418] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 12:43:33 2019][2856733.770349] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 12:43:33 2019][2856733.783752] Lustre: Skipped 1 previous similar message [Thu Apr 11 12:44:12 2019][2856773.000400] LNet: Service thread pid 118527 was inactive for 200.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 12:44:12 2019][2856773.017614] LNet: Skipped 1 previous similar message [Thu Apr 11 12:44:12 2019][2856773.022760] Pid: 118527, comm: mdt00_092 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 12:44:12 2019][2856773.032760] Call Trace: [Thu Apr 11 12:44:12 2019][2856773.035402] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 12:44:12 2019][2856773.042507] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 12:44:12 2019][2856773.049875] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 12:44:12 2019][2856773.056814] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 12:44:12 2019][2856773.063488] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 12:44:12 2019][2856773.069726] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 12:44:12 2019][2856773.076474] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 12:44:12 2019][2856773.082349] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 12:44:12 2019][2856773.089472] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 12:44:12 2019][2856773.097370] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 12:44:12 2019][2856773.103878] [] kthread+0xd1/0xe0 [Thu Apr 11 12:44:12 2019][2856773.108988] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 12:44:12 2019][2856773.115829] [] 0xffffffffffffffff [Thu Apr 11 12:44:12 2019][2856773.121760] LustreError: dumping log to /tmp/lustre-log.1555011852.118527 [Thu Apr 11 12:44:14 2019][2856774.533788] LNet: Service thread pid 117467 was inactive for 202.26s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 12:44:14 2019][2856774.551009] Pid: 117467, comm: mdt01_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 12:44:14 2019][2856774.561008] Call Trace: [Thu Apr 11 12:44:14 2019][2856774.563651] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 12:44:14 2019][2856774.570771] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 12:44:14 2019][2856774.578138] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 12:44:14 2019][2856774.585153] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 12:44:14 2019][2856774.592335] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 12:44:14 2019][2856774.598991] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 12:44:14 2019][2856774.606168] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 12:44:14 2019][2856774.612911] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 12:44:14 2019][2856774.619567] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 12:44:14 2019][2856774.626534] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 12:44:14 2019][2856774.633826] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 12:44:14 2019][2856774.640184] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 12:44:14 2019][2856774.647308] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 12:44:14 2019][2856774.655204] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 12:44:14 2019][2856774.661722] [] kthread+0xd1/0xe0 [Thu Apr 11 12:44:14 2019][2856774.666799] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 12:44:14 2019][2856774.673447] [] 0xffffffffffffffff [Thu Apr 11 12:45:58 2019][2856879.110348] LNetError: 117129:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 12:46:24 2019][2856905.064807] LustreError: 119055:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001083f:0xdcef:0x0] [Thu Apr 11 12:48:02 2019][2857003.090516] Lustre: fir-MDT0002: Connection restored to 0abd045d-746e-a7a6-f1ec-e59a96ee02da (at 10.9.101.64@o2ib4) [Thu Apr 11 12:48:02 2019][2857003.101122] Lustre: Skipped 1786 previous similar messages [Thu Apr 11 12:48:14 2019][2857014.569391] Lustre: fir-MDT0002: Client 6d84db07-946f-3a55-fea2-55904c526e1b (at 10.8.1.21@o2ib6) reconnecting [Thu Apr 11 12:48:14 2019][2857014.579564] Lustre: Skipped 1690 previous similar messages [Thu Apr 11 12:48:20 2019][2857020.609983] LustreError: 118657:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001082f:0x3174:0x0] [Thu Apr 11 12:50:45 2019][2857165.785025] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 12:52:39 2019][2857279.275903] LustreError: 118731:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5ddb2fc850 x1628758129265376/t0(0) o4->91934493-be40-5b23-a558-2943654113ab@10.8.27.16@o2ib6:8/0 lens 488/448 e 0 to 0 dl 1555012388 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 12:52:39 2019][2857279.277168] Lustre: fir-MDT0002: Bulk IO write error with 91934493-be40-5b23-a558-2943654113ab (at 10.8.27.16@o2ib6), client will retry: rc = -107 [Thu Apr 11 12:52:39 2019][2857279.277170] Lustre: Skipped 3 previous similar messages [Thu Apr 11 12:52:39 2019][2857279.319172] LustreError: 118731:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 12 previous similar messages [Thu Apr 11 12:58:03 2019][2857603.375919] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.23@o2ib6) [Thu Apr 11 12:58:03 2019][2857603.383235] Lustre: Skipped 1751 previous similar messages [Thu Apr 11 12:58:14 2019][2857614.912461] Lustre: fir-MDT0002: Client b477f0c2-cfda-70b4-919f-fe195c63921e (at 10.8.7.22@o2ib6) reconnecting [Thu Apr 11 12:58:14 2019][2857614.922646] Lustre: Skipped 1730 previous similar messages [Thu Apr 11 13:01:05 2019][2857785.976238] LustreError: 119137:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b4e8d1f8600 x1628639295107536/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:5/0 lens 448/440 e 0 to 0 dl 1555012895 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 13:01:19 2019][2857799.907802] LustreError: 119020:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001083f:0xdfcd:0x0] [Thu Apr 11 13:02:40 2019][2857881.041693] LustreError: 118630:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5a5a916050 x1629293190883040/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:10/0 lens 488/448 e 0 to 0 dl 1555012990 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 13:02:40 2019][2857881.066262] LustreError: 118630:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 49 previous similar messages [Thu Apr 11 13:02:40 2019][2857881.076032] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 13:02:40 2019][2857881.089430] Lustre: Skipped 3 previous similar messages [Thu Apr 11 13:07:15 2019][2858156.145750] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 13:08:03 2019][2858203.433073] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.16@o2ib6) [Thu Apr 11 13:08:03 2019][2858203.440470] Lustre: Skipped 1685 previous similar messages [Thu Apr 11 13:08:14 2019][2858215.217699] Lustre: fir-MDT0002: Client aa1e9ce5-2ac3-cfae-1629-e9796f53cb3b (at 10.9.107.30@o2ib4) reconnecting [Thu Apr 11 13:08:14 2019][2858215.228053] Lustre: Skipped 1699 previous similar messages [Thu Apr 11 13:10:40 2019][2858360.930639] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 13:13:00 2019][2858500.912825] Lustre: 118155:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 13:13:00 2019][2858500.912825] req@ffff8b75fdbc5a00 x1629290069774928/t0(0) o101->4ad347ca-bc6d-204b-6215-9b20ec9910f0@10.9.102.17@o2ib4:5/0 lens 592/3264 e 0 to 0 dl 1555013585 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 13:13:20 2019][2858521.070533] LustreError: 118027:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b72fc2b3f00 x1628639297069552/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:26/0 lens 448/440 e 0 to 0 dl 1555013606 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 13:13:31 2019][2858532.077817] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 13:13:31 2019][2858532.179742] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 13:14:05 2019][2858565.922135] LustreError: 118400:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555013555, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b6136b71200/0x53d0f21a18bb825b lrc: 3/1,0 mode: --/PR res: [0x2c0011204:0x35aa:0x0].0x0 bits 0x13/0x8 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118400 timeout: 0 lvb_type: 0 [Thu Apr 11 13:14:05 2019][2858565.961966] LustreError: 118400:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 13:15:56 2019][2858676.655409] LNet: Service thread pid 118448 was inactive for 200.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 13:15:56 2019][2858676.672652] Pid: 118448, comm: mdt01_110 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 13:15:56 2019][2858676.682677] Call Trace: [Thu Apr 11 13:15:56 2019][2858676.685330] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 13:15:56 2019][2858676.692450] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 13:15:56 2019][2858676.699842] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 13:15:56 2019][2858676.706766] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 13:15:56 2019][2858676.713443] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 13:15:56 2019][2858676.719670] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 13:15:56 2019][2858676.726562] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 13:15:56 2019][2858676.732644] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 13:15:56 2019][2858676.739787] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 13:15:56 2019][2858676.747683] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 13:15:56 2019][2858676.754183] [] kthread+0xd1/0xe0 [Thu Apr 11 13:15:56 2019][2858676.759296] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 13:15:56 2019][2858676.765945] [] 0xffffffffffffffff [Thu Apr 11 13:15:56 2019][2858676.771395] LustreError: dumping log to /tmp/lustre-log.1555013756.118448 [Thu Apr 11 13:15:57 2019][2858678.163588] LNet: Service thread pid 118400 was inactive for 202.23s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 13:15:57 2019][2858678.180874] Pid: 118400, comm: mdt01_101 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 13:15:57 2019][2858678.190878] Call Trace: [Thu Apr 11 13:15:57 2019][2858678.193515] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 13:15:57 2019][2858678.200630] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 13:15:57 2019][2858678.208013] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 13:15:57 2019][2858678.215016] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 13:15:57 2019][2858678.222192] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 13:15:57 2019][2858678.228848] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 13:15:57 2019][2858678.236017] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 13:15:57 2019][2858678.242761] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 13:15:57 2019][2858678.249414] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 13:15:57 2019][2858678.256348] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 13:15:57 2019][2858678.263627] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 13:15:57 2019][2858678.269974] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 13:15:57 2019][2858678.277088] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 13:15:57 2019][2858678.284985] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 13:15:57 2019][2858678.291484] [] kthread+0xd1/0xe0 [Thu Apr 11 13:15:57 2019][2858678.296572] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 13:15:58 2019][2858678.303220] [] 0xffffffffffffffff [Thu Apr 11 13:18:03 2019][2858803.639352] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Thu Apr 11 13:18:03 2019][2858803.646811] Lustre: Skipped 1722 previous similar messages [Thu Apr 11 13:18:15 2019][2858815.644670] Lustre: fir-MDT0002: Client 369e48ad-81ca-5881-14c8-c80031c22f3c (at 10.9.104.17@o2ib4) reconnecting [Thu Apr 11 13:18:15 2019][2858815.655027] Lustre: Skipped 1719 previous similar messages [Thu Apr 11 13:20:56 2019][2858976.625000] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 13:20:57 2019][2858978.082283] LNetError: 117130:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 13:22:18 2019][2859059.202977] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 13:22:18 2019][2859059.216361] Lustre: Skipped 6 previous similar messages [Thu Apr 11 13:28:03 2019][2859403.890607] Lustre: fir-MDT0002: Connection restored to (at 10.8.26.21@o2ib6) [Thu Apr 11 13:28:03 2019][2859403.898009] Lustre: Skipped 1739 previous similar messages [Thu Apr 11 13:28:15 2019][2859416.272832] Lustre: fir-MDT0002: Client e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6) reconnecting [Thu Apr 11 13:28:15 2019][2859416.283008] Lustre: Skipped 1734 previous similar messages [Thu Apr 11 13:29:33 2019][2859493.334204] LustreError: 119125:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6a842a8850 x1629293194056544/t0(0) o3->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:8/0 lens 488/440 e 0 to 0 dl 1555014578 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 13:29:33 2019][2859493.358661] LustreError: 119125:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 16 previous similar messages [Thu Apr 11 13:29:33 2019][2859493.368431] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 13:38:04 2019][2860004.592765] Lustre: fir-MDT0002: Connection restored to 83b62b04-91a2-2007-7323-cfca5b989386 (at 10.9.103.42@o2ib4) [Thu Apr 11 13:38:04 2019][2860004.603371] Lustre: Skipped 1732 previous similar messages [Thu Apr 11 13:38:15 2019][2860016.293945] Lustre: fir-MDT0002: Client 1bde1f20-37f1-ed48-746c-38acfc0e9a71 (at 10.9.106.55@o2ib4) reconnecting [Thu Apr 11 13:38:15 2019][2860016.304292] Lustre: Skipped 1738 previous similar messages [Thu Apr 11 13:41:57 2019][2860237.491230] LustreError: 119135:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b56f4fae850 x1629293195451776/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:27/0 lens 488/448 e 0 to 0 dl 1555015347 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 13:41:57 2019][2860237.515793] LustreError: 119135:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 3 previous similar messages [Thu Apr 11 13:41:57 2019][2860237.525474] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 13:41:57 2019][2860237.538857] Lustre: Skipped 2 previous similar messages [Thu Apr 11 13:44:44 2019][2860404.348693] Lustre: fir-MDT0001: haven't heard from client cd34f184-137e-6bcb-94d8-161cf6d54a44 (at 10.9.112.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5671b35800, cur 1555015484 expire 1555015334 last 1555015257 [Thu Apr 11 13:44:44 2019][2860404.370575] Lustre: Skipped 4 previous similar messages [Thu Apr 11 13:44:46 2019][2860406.346170] Lustre: fir-MDT0000: haven't heard from client cd34f184-137e-6bcb-94d8-161cf6d54a44 (at 10.9.112.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8c3440dc00, cur 1555015486 expire 1555015336 last 1555015259 [Thu Apr 11 13:48:04 2019][2860604.783897] Lustre: fir-MDT0002: Connection restored to 30d4579a-fbb3-061d-f340-d61cdf6fe8c4 (at 10.9.108.33@o2ib4) [Thu Apr 11 13:48:04 2019][2860604.794522] Lustre: Skipped 1820 previous similar messages [Thu Apr 11 13:48:17 2019][2860618.159296] Lustre: fir-MDT0002: Client b477f0c2-cfda-70b4-919f-fe195c63921e (at 10.8.7.22@o2ib6) reconnecting [Thu Apr 11 13:48:17 2019][2860618.169491] Lustre: Skipped 1816 previous similar messages [Thu Apr 11 13:51:15 2019][2860795.723713] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 13:53:15 2019][2860915.632228] LustreError: 119186:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b53c359b600 x1628639301145312/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:15/0 lens 448/440 e 0 to 0 dl 1555016025 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 13:53:15 2019][2860915.656814] LustreError: 119186:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 114 previous similar messages [Thu Apr 11 13:55:23 2019][2861043.359595] Lustre: fir-MDT0000: haven't heard from client b69a7635-6e49-3b88-f9c4-7acfd5749b13 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b723e660c00, cur 1555016123 expire 1555015973 last 1555015896 [Thu Apr 11 13:55:23 2019][2861043.381516] Lustre: Skipped 3 previous similar messages [Thu Apr 11 13:58:04 2019][2861205.251896] Lustre: fir-MDT0002: Connection restored to (at 10.9.101.72@o2ib4) [Thu Apr 11 13:58:04 2019][2861205.259381] Lustre: Skipped 1764 previous similar messages [Thu Apr 11 13:58:18 2019][2861219.156958] Lustre: fir-MDT0002: Client a132911f-e28d-2e04-1263-47066608ec3c (at 10.8.23.24@o2ib6) reconnecting [Thu Apr 11 13:58:18 2019][2861219.167244] Lustre: Skipped 1760 previous similar messages [Thu Apr 11 13:58:29 2019][2861229.675669] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 14:00:08 2019][2861328.761149] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 14:06:45 2019][2861725.774856] LustreError: 118879:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5a5a911850 x1629293198056128/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:21/0 lens 488/448 e 0 to 0 dl 1555016811 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 14:06:45 2019][2861725.799424] LustreError: 118879:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 8 previous similar messages [Thu Apr 11 14:08:05 2019][2861805.675277] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.15@o2ib6) [Thu Apr 11 14:08:05 2019][2861805.682596] Lustre: Skipped 1731 previous similar messages [Thu Apr 11 14:08:19 2019][2861819.398968] Lustre: fir-MDT0002: Client bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83 (at 10.8.26.21@o2ib6) reconnecting [Thu Apr 11 14:08:19 2019][2861819.409248] Lustre: Skipped 1739 previous similar messages [Thu Apr 11 14:09:51 2019][2861911.764303] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 14:09:51 2019][2861911.777687] Lustre: Skipped 10 previous similar messages [Thu Apr 11 14:10:09 2019][2861930.305935] LNetError: 117122:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 14:10:22 2019][2861942.797921] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 14:16:46 2019][2862327.030238] LustreError: 119079:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b71f2ee0900 x1628639303473040/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:16/0 lens 448/440 e 0 to 0 dl 1555017436 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 14:16:46 2019][2862327.054801] LustreError: 119079:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 107 previous similar messages [Thu Apr 11 14:17:05 2019][2862345.992792] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 14:18:06 2019][2862406.518157] Lustre: fir-MDT0002: Connection restored to a70f6134-3834-8f00-05fc-8137b3a6e8df (at 10.9.102.7@o2ib4) [Thu Apr 11 14:18:06 2019][2862406.528678] Lustre: Skipped 1839 previous similar messages [Thu Apr 11 14:18:20 2019][2862420.500125] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Thu Apr 11 14:18:20 2019][2862420.510216] Lustre: Skipped 1830 previous similar messages [Thu Apr 11 14:28:06 2019][2863006.616719] Lustre: fir-MDT0002: Connection restored to (at 10.9.114.5@o2ib4) [Thu Apr 11 14:28:06 2019][2863006.624129] Lustre: Skipped 1782 previous similar messages [Thu Apr 11 14:28:20 2019][2863020.614541] Lustre: fir-MDT0002: Client f87bfb7b-5313-2d83-7524-cc6a927681f0 (at 10.9.108.65@o2ib4) reconnecting [Thu Apr 11 14:28:20 2019][2863020.624887] Lustre: Skipped 1785 previous similar messages [Thu Apr 11 14:31:49 2019][2863230.284024] LustreError: 119211:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b79bc33bf00 x1628639304990736/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:19/0 lens 448/440 e 0 to 0 dl 1555018339 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 14:31:49 2019][2863230.308600] LustreError: 119211:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 54 previous similar messages [Thu Apr 11 14:32:35 2019][2863276.323163] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 14:35:40 2019][2863461.307741] LustreError: 118086:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b5ad13b5d00 x1628639305377872/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:10/0 lens 448/440 e 0 to 0 dl 1555018570 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 14:38:06 2019][2863606.636454] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 14:38:06 2019][2863606.646984] Lustre: Skipped 1732 previous similar messages [Thu Apr 11 14:38:17 2019][2863617.546213] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 14:38:17 2019][2863617.559588] Lustre: Skipped 4 previous similar messages [Thu Apr 11 14:38:20 2019][2863621.372518] Lustre: fir-MDT0002: Client b477f0c2-cfda-70b4-919f-fe195c63921e (at 10.8.7.22@o2ib6) reconnecting [Thu Apr 11 14:38:20 2019][2863621.382700] Lustre: Skipped 1731 previous similar messages [Thu Apr 11 14:39:19 2019][2863679.553658] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 14:48:06 2019][2864207.096422] Lustre: fir-MDT0003: Connection restored to (at 10.0.10.3@o2ib7) [Thu Apr 11 14:48:06 2019][2864207.103733] Lustre: Skipped 1794 previous similar messages [Thu Apr 11 14:48:21 2019][2864221.554139] Lustre: fir-MDT0002: Client 8edb63b6-d814-54d5-ddee-ddb62f3308b6 (at 10.9.103.42@o2ib4) reconnecting [Thu Apr 11 14:48:21 2019][2864221.564493] Lustre: Skipped 1794 previous similar messages [Thu Apr 11 14:49:39 2019][2864299.724837] LustreError: 118445:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5e17ac2850 x1629293202125280/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:8/0 lens 488/448 e 0 to 0 dl 1555019408 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 14:49:39 2019][2864299.749315] LustreError: 118445:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 69 previous similar messages [Thu Apr 11 14:49:39 2019][2864299.759068] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 14:49:41 2019][2864301.497920] LustreError: 119028:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b59418abc00 x1628639306787040/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:11/0 lens 448/440 e 0 to 0 dl 1555019411 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 14:53:51 2019][2864551.431918] Lustre: fir-MDT0000: haven't heard from client b3a57f6b-5ce4-00c7-65c9-2d00650ee57d (at 10.9.115.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8cf9b7b800, cur 1555019631 expire 1555019481 last 1555019404 [Thu Apr 11 14:53:51 2019][2864551.453891] Lustre: Skipped 4 previous similar messages [Thu Apr 11 14:55:07 2019][2864627.434471] Lustre: fir-MDT0000: haven't heard from client 92f69cbf-9097-b9ad-8103-3dd6ecd89d27 (at 10.8.25.24@o2ib6) in 222 seconds. I think it's dead, and I am evicting it. exp ffff8b886830e400, cur 1555019707 expire 1555019557 last 1555019485 [Thu Apr 11 14:55:07 2019][2864627.456366] Lustre: Skipped 4 previous similar messages [Thu Apr 11 14:55:12 2019][2864632.437174] Lustre: MGS: haven't heard from client 13ce3b4f-fb05-e035-fc29-aecf2679d300 (at 10.8.25.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b898a699400, cur 1555019712 expire 1555019562 last 1555019485 [Thu Apr 11 14:55:12 2019][2864632.458466] Lustre: Skipped 3 previous similar messages [Thu Apr 11 14:58:06 2019][2864807.381341] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.23@o2ib6) [Thu Apr 11 14:58:06 2019][2864807.388673] Lustre: Skipped 1763 previous similar messages [Thu Apr 11 14:58:21 2019][2864821.962511] Lustre: fir-MDT0002: Client eb030a79-9cda-5e09-c87b-4c55bbd9e3db (at 10.9.102.7@o2ib4) reconnecting [Thu Apr 11 14:58:21 2019][2864821.972777] Lustre: Skipped 1766 previous similar messages [Thu Apr 11 15:01:01 2019][2864981.589936] LustreError: 118972:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b62c94a0850 x1629293203108784/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:7/0 lens 488/448 e 0 to 0 dl 1555020067 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 15:01:01 2019][2864981.614454] LustreError: 118972:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 48 previous similar messages [Thu Apr 11 15:01:01 2019][2864981.624241] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 15:01:01 2019][2864981.637671] Lustre: Skipped 6 previous similar messages [Thu Apr 11 15:01:01 2019][2864982.227224] Lustre: 118655:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629293203108832 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b7195fd2c50 x1629293203108832/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:7/0 lens 488/0 e 0 to 0 dl 1555020067 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 15:04:49 2019][2865209.639294] LNetError: 117122:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 15:06:11 2019][2865291.615448] Lustre: fir-MDT0002: Bulk IO read error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc -107 [Thu Apr 11 15:08:07 2019][2865407.613353] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.1@o2ib4) [Thu Apr 11 15:08:07 2019][2865407.620758] Lustre: Skipped 1741 previous similar messages [Thu Apr 11 15:08:21 2019][2865422.030283] Lustre: fir-MDT0002: Client eda11444-0027-aea4-d226-92aced58871b (at 10.9.114.5@o2ib4) reconnecting [Thu Apr 11 15:08:21 2019][2865422.040550] Lustre: Skipped 1744 previous similar messages [Thu Apr 11 15:12:47 2019][2865687.836055] LustreError: 119097:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6f8ba0f500 x1628639309121968/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:17/0 lens 448/440 e 0 to 0 dl 1555020797 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 15:12:47 2019][2865687.860622] LustreError: 119097:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 17 previous similar messages [Thu Apr 11 15:13:49 2019][2865749.456471] Lustre: fir-MDT0000: haven't heard from client 06bc3408-91c6-0bec-9557-cec488729361 (at 10.9.113.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b57c3a36800, cur 1555020829 expire 1555020679 last 1555020602 [Thu Apr 11 15:15:57 2019][2865877.689409] LNetError: 117129:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 15:16:59 2019][2865939.879018] LustreError: 119206:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b53472ab000 x1628639309541856/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:6/0 lens 448/440 e 0 to 0 dl 1555021026 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Apr 11 15:18:07 2019][2866008.146237] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Thu Apr 11 15:18:07 2019][2866008.153639] Lustre: Skipped 1752 previous similar messages [Thu Apr 11 15:18:21 2019][2866022.178170] Lustre: fir-MDT0002: Client 58899832-571e-4a18-664b-6f5c1308e27e (at 10.8.18.18@o2ib6) reconnecting [Thu Apr 11 15:18:21 2019][2866022.188472] Lustre: Skipped 1742 previous similar messages [Thu Apr 11 15:18:48 2019][2866048.726423] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Thu Apr 11 15:19:37 2019][2866097.906416] Lustre: fir-MDT0002: Bulk IO write error with 4cddd28c-e741-4a01-9c91-62f73129c1c5 (at 10.9.104.19@o2ib4), client will retry: rc = -107 [Thu Apr 11 15:19:37 2019][2866097.919861] Lustre: Skipped 9 previous similar messages [Thu Apr 11 15:22:27 2019][2866268.002782] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 15:23:45 2019][2866345.809034] LustreError: 118888:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b8ccd78dc50 x1629293205738768/t0(0) o4->4cddd28c-e741-4a01-9c91-62f73129c1c5@10.9.104.19@o2ib4:21/0 lens 488/448 e 0 to 0 dl 1555021431 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 15:23:45 2019][2866345.833595] LustreError: 118888:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 13 previous similar messages [Thu Apr 11 15:25:16 2019][2866436.470534] Lustre: fir-MDT0000: haven't heard from client b10a1e9e-353e-14e1-cf2c-6ea26e2706ae (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6f63ba1800, cur 1555021516 expire 1555021366 last 1555021289 [Thu Apr 11 15:25:16 2019][2866436.492417] Lustre: Skipped 4 previous similar messages [Thu Apr 11 15:25:22 2019][2866442.472624] Lustre: fir-MDT0001: haven't heard from client b10a1e9e-353e-14e1-cf2c-6ea26e2706ae (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b639a217c00, cur 1555021522 expire 1555021372 last 1555021295 [Thu Apr 11 15:25:22 2019][2866442.494506] Lustre: Skipped 3 previous similar messages [Thu Apr 11 15:28:07 2019][2866608.201707] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Thu Apr 11 15:28:07 2019][2866608.212163] Lustre: Skipped 1750 previous similar messages [Thu Apr 11 15:28:22 2019][2866623.026052] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Thu Apr 11 15:28:22 2019][2866623.036401] Lustre: Skipped 1740 previous similar messages [Thu Apr 11 15:29:35 2019][2866696.002890] LustreError: 119216:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b6cfc4e7050 x1628639311824800/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:5/0 lens 448/440 e 0 to 0 dl 1555021805 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Apr 11 15:32:44 2019][2866885.275810] LustreError: 119108:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b6ac460c800 x1628640535104704/t0(0) o37->f87bfb7b-5313-2d83-7524-cc6a927681f0@10.9.108.65@o2ib4:20/0 lens 448/440 e 0 to 0 dl 1555021970 ref 1 fl Interpret:/2/0 rc 0/0 [Thu Apr 11 15:34:56 2019][2867017.398173] LustreError: 119106:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b678e608c00 x1628870361683136/t0(0) o37->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:2/0 lens 448/440 e 0 to 0 dl 1555022102 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 15:34:56 2019][2867017.422664] LustreError: 119106:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 23 previous similar messages [Thu Apr 11 15:35:32 2019][2867053.076631] LustreError: 119226:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b7530e0e900 x1628639313990000/t0(0) o37->01f19ed0-770e-84b4-7586-12f4195b7925@10.9.108.34@o2ib4:8/0 lens 448/440 e 0 to 0 dl 1555022138 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 15:35:59 2019][2867080.426270] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 15:35:59 2019][2867080.439675] Lustre: Skipped 2 previous similar messages [Thu Apr 11 15:38:08 2019][2867209.229319] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Thu Apr 11 15:38:08 2019][2867209.239685] Lustre: Skipped 1762 previous similar messages [Thu Apr 11 15:38:24 2019][2867224.589840] Lustre: fir-MDT0002: Client 1fed4b9d-ca5d-165d-887c-2f1234134d0c (at 10.8.9.10@o2ib6) reconnecting [Thu Apr 11 15:38:24 2019][2867224.600021] Lustre: Skipped 1760 previous similar messages [Thu Apr 11 15:40:13 2019][2867333.488880] Lustre: fir-MDT0002: haven't heard from client 99d4c24c-ac8e-82f6-f9b9-6461bad9acdd (at 10.9.115.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7759775000, cur 1555022413 expire 1555022263 last 1555022186 [Thu Apr 11 15:44:45 2019][2867605.494489] Lustre: fir-MDT0003: haven't heard from client 773f16f2-6cde-baf5-58f0-06a2222f87b2 (at 10.8.23.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b647afac400, cur 1555022685 expire 1555022535 last 1555022458 [Thu Apr 11 15:44:45 2019][2867605.516373] Lustre: Skipped 4 previous similar messages [Thu Apr 11 15:44:58 2019][2867618.494715] Lustre: fir-MDT0002: haven't heard from client 773f16f2-6cde-baf5-58f0-06a2222f87b2 (at 10.8.23.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7cfefef800, cur 1555022698 expire 1555022548 last 1555022471 [Thu Apr 11 15:44:58 2019][2867618.516596] Lustre: Skipped 7 previous similar messages [Thu Apr 11 15:46:30 2019][2867711.385635] LustreError: 119009:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010621:0xce0a:0x0] [Thu Apr 11 15:47:32 2019][2867773.426244] LustreError: 119186:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5203b05700 x1628870366942464/t0(0) o37->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:8/0 lens 448/440 e 0 to 0 dl 1555022858 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 15:47:33 2019][2867773.450754] LustreError: 119186:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 17 previous similar messages [Thu Apr 11 15:47:50 2019][2867790.656588] LNetError: 117119:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 15:48:08 2019][2867809.273719] Lustre: fir-MDT0002: Connection restored to (at 10.9.101.72@o2ib4) [Thu Apr 11 15:48:08 2019][2867809.281203] Lustre: Skipped 1741 previous similar messages [Thu Apr 11 15:48:24 2019][2867825.351660] Lustre: fir-MDT0002: Client ee5f7d3d-8ffe-7ca2-e4f8-dd26c84b3cb6 (at 10.9.107.48@o2ib4) reconnecting [Thu Apr 11 15:48:24 2019][2867825.362058] Lustre: Skipped 1736 previous similar messages [Thu Apr 11 15:48:57 2019][2867857.481785] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 15:49:18 2019][2867878.472216] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 15:49:18 2019][2867878.485600] Lustre: Skipped 6 previous similar messages [Thu Apr 11 15:54:08 2019][2868169.439294] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 15:58:09 2019][2868409.550931] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.15@o2ib6) [Thu Apr 11 15:58:09 2019][2868409.558275] Lustre: Skipped 1740 previous similar messages [Thu Apr 11 15:58:24 2019][2868424.604960] LustreError: 119096:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b761e741200 x1628870369775616/t0(0) o37->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:0/0 lens 448/440 e 0 to 0 dl 1555023510 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 15:58:24 2019][2868424.629547] LustreError: 119096:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 21 previous similar messages [Thu Apr 11 15:58:25 2019][2868425.570838] Lustre: fir-MDT0002: Client eda11444-0027-aea4-d226-92aced58871b (at 10.9.114.5@o2ib4) reconnecting [Thu Apr 11 15:58:25 2019][2868425.581116] Lustre: Skipped 1748 previous similar messages [Thu Apr 11 15:59:30 2019][2868491.029445] LNetError: 117129:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Thu Apr 11 15:59:48 2019][2868508.665533] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 16:00:19 2019][2868539.748726] Lustre: fir-MDT0002: Bulk IO write error with 10601291-a183-8deb-41a7-af6620223334 (at 10.9.108.36@o2ib4), client will retry: rc = -107 [Thu Apr 11 16:00:19 2019][2868539.762153] Lustre: Skipped 2 previous similar messages [Thu Apr 11 16:07:09 2019][2868949.726729] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 16:07:09 2019][2868949.739861] Lustre: Skipped 1 previous similar message [Thu Apr 11 16:08:09 2019][2869009.868622] Lustre: fir-MDT0002: Connection restored to 3f1d1974-7ed8-ffd0-1256-fba8ddd54f53 (at 10.8.9.10@o2ib6) [Thu Apr 11 16:08:09 2019][2869009.879059] Lustre: Skipped 1748 previous similar messages [Thu Apr 11 16:08:25 2019][2869025.756534] Lustre: fir-MDT0002: Client f53d399b-a6b5-36d0-034f-8c8c0efbb65f (at 10.9.106.11@o2ib4) reconnecting [Thu Apr 11 16:08:25 2019][2869025.766932] Lustre: Skipped 1742 previous similar messages [Thu Apr 11 16:08:54 2019][2869054.829817] LustreError: 119081:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b722c2b0c00 x1628870371989744/t0(0) o37->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:0/0 lens 448/440 e 0 to 0 dl 1555024140 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:08:54 2019][2869054.854572] LustreError: 119081:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 11 previous similar messages [Thu Apr 11 16:12:24 2019][2869264.821107] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 16:12:24 2019][2869264.834489] Lustre: Skipped 1 previous similar message [Thu Apr 11 16:15:09 2019][2869430.389694] LustreError: 118425:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b2:0x94b:0x0] [Thu Apr 11 16:15:33 2019][2869453.857689] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 16:18:10 2019][2869610.861643] Lustre: fir-MDT0002: Connection restored to a70f6134-3834-8f00-05fc-8137b3a6e8df (at 10.9.102.7@o2ib4) [Thu Apr 11 16:18:10 2019][2869610.872176] Lustre: Skipped 1747 previous similar messages [Thu Apr 11 16:18:25 2019][2869625.806484] Lustre: fir-MDT0002: Client 35ddd205-c174-6b95-20ef-c38a3193733f (at 10.9.108.35@o2ib4) reconnecting [Thu Apr 11 16:18:25 2019][2869625.816843] Lustre: Skipped 1734 previous similar messages [Thu Apr 11 16:19:03 2019][2869663.815503] LustreError: 119241:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b52f7a2e600 x1628870373800528/t0(0) o37->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:9/0 lens 448/440 e 0 to 0 dl 1555024749 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:19:03 2019][2869663.843693] LustreError: 119241:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 8 previous similar messages [Thu Apr 11 16:20:24 2019][2869744.691311] Lustre: 118317:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 16:20:24 2019][2869744.691311] req@ffff8b6886efbf00 x1629300786222848/t0(0) o101->74806a48-c1e8-fa00-f93a-73121ee183ec@10.9.103.43@o2ib4:29/0 lens 576/3264 e 1 to 0 dl 1555024829 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:20:24 2019][2869744.720603] Lustre: 118317:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 16:20:24 2019][2869745.237266] Lustre: 118191:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 16:20:24 2019][2869745.237266] req@ffff8b71eef6d400 x1629858364200320/t146353589066(0) o36->f419f2f7-2340-142c-2553-5bd564397f99@10.8.28.6@o2ib6:29/0 lens 544/2888 e 1 to 0 dl 1555024829 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:20:45 2019][2869765.574661] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 16:20:45 2019][2869765.587290] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages [Thu Apr 11 16:21:39 2019][2869819.792794] LustreError: 118370:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555024809, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b693f2b6540/0x53d0f21b08c2835c lrc: 3/1,0 mode: --/PR res: [0x2c00120cf:0xeaa:0x0].0x0 bits 0x13/0x8 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118370 timeout: 0 lvb_type: 0 [Thu Apr 11 16:21:39 2019][2869819.832656] LustreError: 118370:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 16:23:29 2019][2869930.135050] LNet: Service thread pid 118438 was inactive for 200.34s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 16:23:29 2019][2869930.152263] Pid: 118438, comm: mdt02_056 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 16:23:29 2019][2869930.162267] Call Trace: [Thu Apr 11 16:23:29 2019][2869930.164919] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 16:23:29 2019][2869930.172103] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 16:23:29 2019][2869930.179484] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 16:23:29 2019][2869930.186424] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 16:23:29 2019][2869930.193096] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 16:23:29 2019][2869930.199333] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 16:23:29 2019][2869930.206076] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 16:23:29 2019][2869930.211955] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 16:23:29 2019][2869930.222648] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 16:23:29 2019][2869930.230895] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 16:23:29 2019][2869930.237511] [] kthread+0xd1/0xe0 [Thu Apr 11 16:23:29 2019][2869930.242601] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 16:23:29 2019][2869930.249285] [] 0xffffffffffffffff [Thu Apr 11 16:23:29 2019][2869930.254510] LustreError: dumping log to /tmp/lustre-log.1555025009.118438 [Thu Apr 11 16:23:31 2019][2869931.793871] LNet: Service thread pid 118370 was inactive for 201.99s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 16:23:31 2019][2869931.811103] Pid: 118370, comm: mdt01_092 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 16:23:31 2019][2869931.821111] Call Trace: [Thu Apr 11 16:23:31 2019][2869931.823749] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 16:23:31 2019][2869931.830861] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 16:23:31 2019][2869931.838239] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 16:23:31 2019][2869931.845248] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 16:23:31 2019][2869931.852425] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 16:23:31 2019][2869931.859080] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 16:23:31 2019][2869931.866257] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 16:23:31 2019][2869931.873000] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 16:23:31 2019][2869931.879656] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 16:23:31 2019][2869931.886599] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 16:23:31 2019][2869931.893906] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 16:23:31 2019][2869931.900256] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 16:23:31 2019][2869931.907384] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 16:23:31 2019][2869931.915296] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 16:23:31 2019][2869931.921815] [] kthread+0xd1/0xe0 [Thu Apr 11 16:23:31 2019][2869931.926914] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 16:23:31 2019][2869931.933589] [] 0xffffffffffffffff [Thu Apr 11 16:27:19 2019][2870159.546875] Lustre: fir-MDT0001: haven't heard from client afe212fc-76c9-fad1-e3f7-9a1563e29ebe (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6b3ea76000, cur 1555025239 expire 1555025089 last 1555025012 [Thu Apr 11 16:27:19 2019][2870159.568761] Lustre: Skipped 1 previous similar message [Thu Apr 11 16:27:21 2019][2870161.546830] Lustre: fir-MDT0000: haven't heard from client afe212fc-76c9-fad1-e3f7-9a1563e29ebe (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7037247400, cur 1555025241 expire 1555025091 last 1555025014 [Thu Apr 11 16:28:10 2019][2870211.208295] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.15@o2ib4) [Thu Apr 11 16:28:10 2019][2870211.215799] Lustre: Skipped 1782 previous similar messages [Thu Apr 11 16:28:25 2019][2870226.234795] Lustre: fir-MDT0002: Client 83b8ff7e-e264-e3fb-e7eb-42bd933886ea (at 10.8.1.23@o2ib6) reconnecting [Thu Apr 11 16:28:25 2019][2870226.244978] Lustre: Skipped 1786 previous similar messages [Thu Apr 11 16:28:47 2019][2870247.967188] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 16:29:45 2019][2870305.670774] Lustre: 118434:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 16:29:45 2019][2870305.670774] req@ffff8b8215e74e00 x1629297984079136/t0(0) o101->c1922c67-e67d-59f1-51e4-23d63728d4c0@10.9.103.10@o2ib4:20/0 lens 576/3264 e 1 to 0 dl 1555025390 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:30:57 2019][2870377.929136] LustreError: 118665:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b8c5296e050 x1628870375526288/t0(0) o4->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:27/0 lens 488/448 e 0 to 0 dl 1555025487 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:30:57 2019][2870377.953711] LustreError: 118665:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 4 previous similar messages [Thu Apr 11 16:30:57 2019][2870377.963385] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 16:30:57 2019][2870377.976760] Lustre: Skipped 1 previous similar message [Thu Apr 11 16:31:00 2019][2870381.146289] LustreError: 118782:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555025370, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b8cc6e90480/0x53d0f21b158bc158 lrc: 3/1,0 mode: --/PR res: [0x2c00111c4:0x1e5b:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118782 timeout: 0 lvb_type: 0 [Thu Apr 11 16:31:00 2019][2870381.186033] LustreError: 118782:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 16:32:00 2019][2870440.951805] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 16:32:51 2019][2870491.810556] LNet: Service thread pid 118361 was inactive for 200.66s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 16:32:51 2019][2870491.827771] Pid: 118361, comm: mdt00_062 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 16:32:51 2019][2870491.837779] Call Trace: [Thu Apr 11 16:32:51 2019][2870491.840418] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 16:32:51 2019][2870491.847554] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 16:32:51 2019][2870491.854931] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 16:32:51 2019][2870491.861865] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 16:32:51 2019][2870491.868519] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 16:32:51 2019][2870491.874743] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 16:32:51 2019][2870491.881496] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 16:32:51 2019][2870491.887406] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 16:32:51 2019][2870491.894547] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 16:32:51 2019][2870491.902443] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 16:32:51 2019][2870491.908950] [] kthread+0xd1/0xe0 [Thu Apr 11 16:32:51 2019][2870491.914040] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 16:32:51 2019][2870491.920690] [] 0xffffffffffffffff [Thu Apr 11 16:32:51 2019][2870491.925889] LustreError: dumping log to /tmp/lustre-log.1555025571.118361 [Thu Apr 11 16:32:52 2019][2870493.247391] Pid: 118782, comm: mdt03_106 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 16:32:52 2019][2870493.257406] Call Trace: [Thu Apr 11 16:32:52 2019][2870493.260049] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 16:32:52 2019][2870493.267185] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 16:32:52 2019][2870493.274572] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 16:32:52 2019][2870493.281582] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 16:32:52 2019][2870493.289034] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 16:32:52 2019][2870493.297721] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 16:32:52 2019][2870493.304896] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 16:32:52 2019][2870493.311639] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 16:32:52 2019][2870493.318296] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 16:32:52 2019][2870493.325260] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 16:32:52 2019][2870493.332567] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 16:32:52 2019][2870493.338914] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 16:32:52 2019][2870493.346028] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 16:32:52 2019][2870493.353923] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 16:32:52 2019][2870493.360431] [] kthread+0xd1/0xe0 [Thu Apr 11 16:32:52 2019][2870493.365520] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 16:32:52 2019][2870493.372169] [] 0xffffffffffffffff [Thu Apr 11 16:34:06 2019][2870566.984835] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 16:36:24 2019][2870704.875551] LustreError: 119064:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011211:0x2c32:0x0] [Thu Apr 11 16:36:25 2019][2870706.224490] LustreError: 119021:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001191d:0x1761:0x0] [Thu Apr 11 16:38:10 2019][2870811.235330] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 16:38:10 2019][2870811.245861] Lustre: Skipped 1847 previous similar messages [Thu Apr 11 16:38:26 2019][2870826.585335] Lustre: fir-MDT0002: Client 91934493-be40-5b23-a558-2943654113ab (at 10.8.27.16@o2ib6) reconnecting [Thu Apr 11 16:38:26 2019][2870826.595604] Lustre: Skipped 1838 previous similar messages [Thu Apr 11 16:38:37 2019][2870837.758955] Lustre: fir-MDT0002: Bulk IO read error with 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4), client will retry: rc -107 [Thu Apr 11 16:41:06 2019][2870986.957555] LustreError: 118729:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8b5984463c50 x1628870376937792/t0(0) o3->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:12/0 lens 488/440 e 0 to 0 dl 1555026072 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:41:06 2019][2870986.981838] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -110 [Thu Apr 11 16:41:48 2019][2871029.155574] LustreError: 118623:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b87943f8450 x1628870377038928/t0(0) o4->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:18/0 lens 488/448 e 0 to 0 dl 1555026138 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:41:48 2019][2871029.180137] LustreError: 118623:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 6 previous similar messages [Thu Apr 11 16:41:48 2019][2871029.189825] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 16:41:48 2019][2871029.203204] Lustre: Skipped 2 previous similar messages [Thu Apr 11 16:42:18 2019][2871059.489835] LustreError: 119064:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010615:0x825:0x0] [Thu Apr 11 16:42:51 2019][2871091.978530] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 16:44:05 2019][2871165.780694] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 16:44:07 2019][2871167.618828] LustreError: 119204:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00111e1:0x49e7:0x0] [Thu Apr 11 16:46:41 2019][2871321.569397] Lustre: fir-MDT0000: haven't heard from client 2be46a13-61ed-628e-4fc9-15052dbdf1ed (at 10.8.23.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b58d7a5c000, cur 1555026401 expire 1555026251 last 1555026174 [Thu Apr 11 16:46:41 2019][2871321.591245] Lustre: Skipped 3 previous similar messages [Thu Apr 11 16:48:10 2019][2871411.277140] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.35@o2ib4) [Thu Apr 11 16:48:10 2019][2871411.284667] Lustre: Skipped 1842 previous similar messages [Thu Apr 11 16:48:26 2019][2871426.616843] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Thu Apr 11 16:48:26 2019][2871426.627111] Lustre: Skipped 1848 previous similar messages [Thu Apr 11 16:50:10 2019][2871530.644490] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Thu Apr 11 16:50:10 2019][2871530.657298] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages [Thu Apr 11 16:50:11 2019][2871532.475567] LustreError: 117517:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011182:0xf3fd:0x0] [Thu Apr 11 16:50:16 2019][2871536.662869] LustreError: 118965:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00108bb:0xbfd7:0x0] [Thu Apr 11 16:52:07 2019][2871648.284918] LustreError: 118956:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b7a55b0b850 x1628641542829248/t0(0) o3->10601291-a183-8deb-41a7-af6620223334@10.9.108.36@o2ib4:7/0 lens 488/440 e 0 to 0 dl 1555026757 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 16:52:07 2019][2871648.292256] Lustre: fir-MDT0002: Bulk IO read error with 10601291-a183-8deb-41a7-af6620223334 (at 10.9.108.36@o2ib4), client will retry: rc -107 [Thu Apr 11 16:52:07 2019][2871648.322436] LustreError: 118956:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 7 previous similar messages [Thu Apr 11 16:54:24 2019][2871784.982769] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 16:54:24 2019][2871784.996219] Lustre: Skipped 5 previous similar messages [Thu Apr 11 16:58:11 2019][2872011.551204] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.23@o2ib6) [Thu Apr 11 16:58:11 2019][2872011.558524] Lustre: Skipped 1854 previous similar messages [Thu Apr 11 16:58:26 2019][2872027.082951] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Thu Apr 11 16:58:26 2019][2872027.093142] Lustre: Skipped 1855 previous similar messages [Thu Apr 11 17:01:12 2019][2872193.116758] LustreError: 118845:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011227:0x17a03:0x0] [Thu Apr 11 17:02:32 2019][2872272.938091] LNetError: 117122:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 17:04:29 2019][2872390.062373] LustreError: 118763:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010825:0xbfd0:0x0] [Thu Apr 11 17:05:36 2019][2872457.189377] LustreError: 118940:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5f59259c50 x1628870380143088/t0(0) o4->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:6/0 lens 488/448 e 0 to 0 dl 1555027566 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:05:36 2019][2872457.213871] LustreError: 118940:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 6 previous similar messages [Thu Apr 11 17:05:36 2019][2872457.223548] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 17:05:36 2019][2872457.236943] Lustre: Skipped 1 previous similar message [Thu Apr 11 17:06:18 2019][2872499.165615] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 17:06:18 2019][2872499.178732] Lustre: Skipped 1 previous similar message [Thu Apr 11 17:06:18 2019][2872499.371611] Lustre: 118309:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:06:18 2019][2872499.371611] req@ffff8b678b6f5700 x1628823115925040/t0(0) o101->58b75192-b975-3fa8-f93d-47a68dd1093f@10.9.103.14@o2ib4:23/0 lens 576/3264 e 0 to 0 dl 1555027583 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:06:18 2019][2872499.400984] Lustre: 118309:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 17:07:09 2019][2872550.409743] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 17:07:23 2019][2872564.549834] LustreError: 118155:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555027553, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b59e8e9dc40/0x53d0f21b4c2803a5 lrc: 3/1,0 mode: --/PR res: [0x2c00106ba:0x21e9:0x0].0x0 bits 0x13/0x8 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118155 timeout: 0 lvb_type: 0 [Thu Apr 11 17:07:24 2019][2872564.589667] LustreError: 118155:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 17:08:11 2019][2872611.640354] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.18@o2ib6) [Thu Apr 11 17:08:11 2019][2872611.647944] Lustre: Skipped 1856 previous similar messages [Thu Apr 11 17:08:26 2019][2872627.096575] Lustre: fir-MDT0002: Client 39b26b25-3151-384d-34d9-da20c3dc7f78 (at 10.8.7.35@o2ib6) reconnecting [Thu Apr 11 17:08:26 2019][2872627.106811] Lustre: Skipped 1854 previous similar messages [Thu Apr 11 17:09:14 2019][2872675.018947] LNet: Service thread pid 118170 was inactive for 200.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 17:09:14 2019][2872675.036143] LNet: Skipped 1 previous similar message [Thu Apr 11 17:09:14 2019][2872675.041302] Pid: 118170, comm: mdt01_017 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:09:14 2019][2872675.051299] Call Trace: [Thu Apr 11 17:09:14 2019][2872675.053942] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 17:09:14 2019][2872675.061081] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 17:09:14 2019][2872675.068456] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 17:09:14 2019][2872675.075380] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 17:09:14 2019][2872675.082037] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 17:09:14 2019][2872675.088259] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 17:09:14 2019][2872675.095021] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 17:09:14 2019][2872675.100895] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:09:14 2019][2872675.108020] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:09:14 2019][2872675.115925] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:09:14 2019][2872675.122441] [] kthread+0xd1/0xe0 [Thu Apr 11 17:09:14 2019][2872675.127529] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:09:14 2019][2872675.134176] [] 0xffffffffffffffff [Thu Apr 11 17:09:14 2019][2872675.139372] LustreError: dumping log to /tmp/lustre-log.1555027754.118170 [Thu Apr 11 17:09:15 2019][2872676.574780] LNet: Service thread pid 118155 was inactive for 202.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 17:09:16 2019][2872676.591976] Pid: 118155, comm: mdt01_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:09:16 2019][2872676.601981] Call Trace: [Thu Apr 11 17:09:16 2019][2872676.604620] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 17:09:16 2019][2872676.611749] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 17:09:16 2019][2872676.619123] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 17:09:16 2019][2872676.626126] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 17:09:16 2019][2872676.633300] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 17:09:16 2019][2872676.639957] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 17:09:16 2019][2872676.647133] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 17:09:16 2019][2872676.653878] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 17:09:16 2019][2872676.660532] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 17:09:16 2019][2872676.667475] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 17:09:16 2019][2872676.674763] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 17:09:16 2019][2872676.681108] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:09:16 2019][2872676.688223] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:09:16 2019][2872676.696118] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:09:16 2019][2872676.702627] [] kthread+0xd1/0xe0 [Thu Apr 11 17:09:16 2019][2872676.707714] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:09:16 2019][2872676.714363] [] 0xffffffffffffffff [Thu Apr 11 17:11:52 2019][2872832.810367] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 17:18:11 2019][2873212.004416] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Thu Apr 11 17:18:11 2019][2873212.011866] Lustre: Skipped 1891 previous similar messages [Thu Apr 11 17:18:26 2019][2873227.123492] Lustre: fir-MDT0002: Client 6d84db07-946f-3a55-fea2-55904c526e1b (at 10.8.1.21@o2ib6) reconnecting [Thu Apr 11 17:18:26 2019][2873227.133686] Lustre: Skipped 1888 previous similar messages [Thu Apr 11 17:18:33 2019][2873234.146478] LustreError: 118729:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5a5a910c50 x1628870381705552/t0(0) o4->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:9/0 lens 488/448 e 0 to 0 dl 1555028319 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:18:33 2019][2873234.170953] LustreError: 118729:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 10 previous similar messages [Thu Apr 11 17:18:33 2019][2873234.180749] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 17:18:33 2019][2873234.194128] Lustre: Skipped 4 previous similar messages [Thu Apr 11 17:24:23 2019][2873584.063974] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 17:25:33 2019][2873654.150088] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 17:25:33 2019][2873654.163228] Lustre: Skipped 1 previous similar message [Thu Apr 11 17:26:13 2019][2873694.294230] Lustre: fir-MDT0002: Bulk IO read error with 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4), client will retry: rc -107 [Thu Apr 11 17:26:32 2019][2873713.384761] Lustre: 118347:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555028785/real 1555028785] req@ffff8b74193c8f00 x1630441151048992/t0(0) o106->fir-MDT0002@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555028792 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Thu Apr 11 17:26:32 2019][2873713.412278] Lustre: 118347:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [Thu Apr 11 17:26:34 2019][2873715.310384] Lustre: 119232:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629298317031648 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b61e1f93c50 x1629298317031648/t0(0) o4->95f309fb-7ae1-b5a5-cb40-881c2341a31d@10.9.106.64@o2ib4:4/0 lens 488/0 e 0 to 0 dl 1555028824 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 17:26:43 2019][2873724.387371] Lustre: 118518:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555028796/real 1555028796] req@ffff8b752d3afb00 x1630441151129392/t0(0) o106->fir-MDT0002@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555028803 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 17:26:43 2019][2873724.414891] Lustre: 118518:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Thu Apr 11 17:26:48 2019][2873729.380085] Lustre: 117531:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 17:26:48 2019][2873729.380085] req@ffff8b710fe0f800 x1628559220007696/t0(0) o101->3c3bbb00-493c-e0ea-5625-2178da530699@10.8.2.32@o2ib6:23/0 lens 480/568 e 1 to 0 dl 1555028813 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:26:48 2019][2873729.409100] Lustre: 117531:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 17:26:50 2019][2873731.384125] Lustre: 118488:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:26:50 2019][2873731.384125] req@ffff8b7cceaf8900 x1628758861264432/t0(0) o101->3cbd02d1-fb8b-33e5-1f1f-b781245520c7@10.9.104.53@o2ib4:25/0 lens 480/568 e 0 to 0 dl 1555028815 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:26:54 2019][2873735.392205] Lustre: 118171:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:26:54 2019][2873735.392205] req@ffff8b78bc248600 x1630377455227728/t0(0) o101->3a3472aa-1f93-3376-3172-e1f39b5cdfe5@10.8.15.9@o2ib6:29/0 lens 480/568 e 0 to 0 dl 1555028819 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:26:56 2019][2873737.396249] Lustre: 117532:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:26:56 2019][2873737.396249] req@ffff8b797f5d4500 x1628570001723152/t0(0) o101->e98ad6a7-830d-1725-2883-db155e5b9e43@10.8.17.16@o2ib6:1/0 lens 480/568 e 0 to 0 dl 1555028821 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:27:00 2019][2873741.424328] Lustre: 118347:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555028813/real 1555028813] req@ffff8b74193c8f00 x1630441151048992/t0(0) o106->fir-MDT0002@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555028820 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 17:27:00 2019][2873741.451869] Lustre: 118347:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [Thu Apr 11 17:27:05 2019][2873746.420431] Lustre: 118431:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:27:05 2019][2873746.420431] req@ffff8b71f2bef500 x1629300966123712/t0(0) o101->d5c505f3-5940-e749-7cb2-d0afce2646b6@10.9.108.6@o2ib4:10/0 lens 480/568 e 0 to 0 dl 1555028830 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:27:16 2019][2873757.210651] Lustre: 118200:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:27:16 2019][2873757.210651] req@ffff8b56c0f5f500 x1630377051661568/t0(0) o101->2b8dc173-d95a-d8da-2099-91594a1e20ba@10.8.7.33@o2ib6:21/0 lens 480/568 e 0 to 0 dl 1555028841 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:27:16 2019][2873757.239983] Lustre: 118200:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 17:27:32 2019][2873773.425976] Lustre: 118518:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555028845/real 1555028845] req@ffff8b752d3afb00 x1630441151129392/t0(0) o106->fir-MDT0002@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555028852 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 17:27:32 2019][2873773.453490] Lustre: 118518:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 70 previous similar messages [Thu Apr 11 17:27:34 2019][2873774.653001] Lustre: 118271:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 17:27:34 2019][2873774.653001] req@ffff8b67a0294e00 x1629291720979248/t0(0) o101->8744cdfa-58ed-088e-3ed1-dc6a4502ac32@10.9.104.30@o2ib4:9/0 lens 592/3264 e 1 to 0 dl 1555028859 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:27:34 2019][2873774.682280] Lustre: 118271:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 5 previous similar messages [Thu Apr 11 17:28:06 2019][2873806.813652] Lustre: 118519:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:28:06 2019][2873806.813652] req@ffff8b773a257200 x1628873391108848/t0(0) o101->b794c6d6-0e55-241b-2244-f3d71f34f2a7@10.9.108.14@o2ib4:11/0 lens 576/3264 e 0 to 0 dl 1555028891 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:28:06 2019][2873806.842982] Lustre: 118519:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 109 previous similar messages [Thu Apr 11 17:28:11 2019][2873812.072999] Lustre: fir-MDT0002: Connection restored to 71f6346b-6c43-76cc-ee66-27a0e00ec0ce (at 10.9.103.8@o2ib4) [Thu Apr 11 17:28:11 2019][2873812.083523] Lustre: Skipped 1993 previous similar messages [Thu Apr 11 17:28:26 2019][2873827.196129] Lustre: fir-MDT0000: Client 70ef6e44-4b1c-8255-8845-5cc3a5ab05ff (at 10.9.107.64@o2ib4) reconnecting [Thu Apr 11 17:28:26 2019][2873827.206474] Lustre: Skipped 2542 previous similar messages [Thu Apr 11 17:28:36 2019][2873837.480267] Lustre: 118468:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555028909/real 1555028909] req@ffff8b56b893f500 x1630441151601536/t0(0) o106->fir-MDT0002@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555028916 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 17:28:36 2019][2873837.507795] Lustre: 118468:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 242 previous similar messages [Thu Apr 11 17:28:49 2019][2873850.039526] LustreError: 118182:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555028839, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b6b038e2400/0x53d0f21b69847a5c lrc: 3/1,0 mode: --/PR res: [0x2000105fc:0xa072:0x0].0x0 bits 0x13/0x0 rrc: 108 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118182 timeout: 0 lvb_type: 0 [Thu Apr 11 17:29:05 2019][2873865.720853] LustreError: 118363:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555028855, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b62c92b3600/0x53d0f21b69d01ce8 lrc: 3/1,0 mode: --/PR res: [0x2c00128dc:0x315a:0x0].0x0 bits 0x13/0x0 rrc: 869 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118363 timeout: 0 lvb_type: 0 [Thu Apr 11 17:29:06 2019][2873866.759874] LustreError: 118236:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555028856, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b8c95637080/0x53d0f21b69d48fdb lrc: 3/1,0 mode: --/PR res: [0x2c00128dc:0x315a:0x0].0x0 bits 0x13/0x0 rrc: 871 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118236 timeout: 0 lvb_type: 0 [Thu Apr 11 17:29:06 2019][2873866.799773] LustreError: 118236:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 19 previous similar messages [Thu Apr 11 17:29:08 2019][2873868.842911] LustreError: 117466:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555028858, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b63f4792ac0/0x53d0f21b69dd8baf lrc: 3/1,0 mode: --/PR res: [0x2c00128dc:0x315a:0x0].0x0 bits 0x13/0x0 rrc: 871 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 117466 timeout: 0 lvb_type: 0 [Thu Apr 11 17:29:08 2019][2873868.882836] LustreError: 117466:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 47 previous similar messages [Thu Apr 11 17:29:10 2019][2873871.326977] Lustre: 118366:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:29:10 2019][2873871.326977] req@ffff8b6b46a93f00 x1630476541860448/t0(0) o101->d5d458f9-7ce3-b81c-9ef0-54b033c5c0e9@10.9.114.2@o2ib4:15/0 lens 1768/0 e 0 to 0 dl 1555028955 ref 2 fl New:/0/ffffffff rc 0/-1 [Thu Apr 11 17:29:10 2019][2873871.356219] Lustre: 118366:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2396 previous similar messages [Thu Apr 11 17:29:12 2019][2873872.862993] LustreError: 118237:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555028862, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b6921a25c40/0x53d0f21b69f04106 lrc: 3/1,0 mode: --/PR res: [0x2c00128dc:0x315a:0x0].0x0 bits 0x13/0x0 rrc: 871 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118237 timeout: 0 lvb_type: 0 [Thu Apr 11 17:29:12 2019][2873872.902897] LustreError: 118237:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 69 previous similar messages [Thu Apr 11 17:29:21 2019][2873881.865182] LustreError: 118510:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555028871, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b6ee5b55100/0x53d0f21b69f49314 lrc: 3/1,0 mode: --/PR res: [0x2000105fc:0xa072:0x0].0x0 bits 0x13/0x0 rrc: 107 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118510 timeout: 0 lvb_type: 0 [Thu Apr 11 17:29:21 2019][2873881.905110] LustreError: 118510:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 22 previous similar messages [Thu Apr 11 17:29:26 2019][2873886.643436] Lustre: fir-MDT0003: haven't heard from client 1b61c2a5-63d9-36db-d1df-22db8107d60d (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6351e3f000, cur 1555028966 expire 1555028816 last 1555028739 [Thu Apr 11 17:29:26 2019][2873886.665312] Lustre: Skipped 4 previous similar messages [Thu Apr 11 17:29:26 2019][2873886.841298] LustreError: 118149:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) failed to reply to blocking AST (req@ffff8b6bb5f2d100 x1630441151610880 status 0 rc -110), evict it ns: mdt-fir-MDT0000_UUID lock: ffff8b59402d60c0/0x53d0f21b67eb4f27 lrc: 4/0,0 mode: PR/PR res: [0x2000105fc:0xa072:0x0].0x0 bits 0x13/0x0 rrc: 107 type: IBT flags: 0x60200400000020 nid: 10.8.27.23@o2ib6 remote: 0x3a9b588cc83988e5 expref: 2499 pid: 118276 timeout: 2873964 lvb_type: 0 [Thu Apr 11 17:29:26 2019][2873886.884776] LustreError: 138-a: fir-MDT0000: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [Thu Apr 11 17:29:26 2019][2873886.897488] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.27.23@o2ib6 ns: mdt-fir-MDT0000_UUID lock: ffff8b59402d60c0/0x53d0f21b67eb4f27 lrc: 3/0,0 mode: PR/PR res: [0x2000105fc:0xa072:0x0].0x0 bits 0x13/0x0 rrc: 107 type: IBT flags: 0x60200400000020 nid: 10.8.27.23@o2ib6 remote: 0x3a9b588cc83988e5 expref: 2500 pid: 118276 timeout: 0 lvb_type: 0 [Thu Apr 11 17:29:26 2019][2873886.951532] Lustre: 118572:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (61:1s); client may timeout. req@ffff8b595f0c4e00 x1629035100990368/t0(0) o101->36e2649c-5bde-7360-b2cc-505510b79efe@10.8.7.26@o2ib6:24/0 lens 576/536 e 0 to 0 dl 1555028965 ref 1 fl Complete:/0/0 rc 0/0 [Thu Apr 11 17:29:26 2019][2873886.962997] LustreError: 118389:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.101.12@o2ib4: deadline 30:6s ago [Thu Apr 11 17:29:26 2019][2873886.962997] req@ffff8b5fd6211b00 x1628755413708016/t0(0) o101->a487b03f-d5d2-c729-8ddf-d69e191e351d@10.9.101.12@o2ib4:20/0 lens 480/0 e 0 to 0 dl 1555028960 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 17:29:26 2019][2873887.012261] Lustre: 118572:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 2589 previous similar messages [Thu Apr 11 17:29:38 2019][2873898.817528] LustreError: 118157:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555028888, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b65f6709d40/0x53d0f21b69f4bb3f lrc: 3/1,0 mode: --/PR res: [0x2c00128dc:0x315a:0x0].0x0 bits 0x13/0x0 rrc: 880 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118157 timeout: 0 lvb_type: 0 [Thu Apr 11 17:29:38 2019][2873898.857433] LustreError: 118157:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 12 previous similar messages [Thu Apr 11 17:29:40 2019][2873900.668719] Lustre: MGS: haven't heard from client 772bda03-b70b-1f60-b191-46f3a384dc97 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8a7ffc3000, cur 1555028980 expire 1555028830 last 1555028753 [Thu Apr 11 17:29:40 2019][2873900.689911] Lustre: Skipped 1 previous similar message [Thu Apr 11 17:29:46 2019][2873906.915691] LNet: Service thread pid 118347 was inactive for 200.52s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 17:29:46 2019][2873906.932884] Pid: 118347, comm: mdt02_041 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:29:46 2019][2873906.942886] Call Trace: [Thu Apr 11 17:29:46 2019][2873906.945533] [] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873906.952332] [] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873906.959184] [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873906.966118] [] mdt_do_glimpse+0x1e9/0x4c0 [mdt] [Thu Apr 11 17:29:46 2019][2873906.972521] [] mdt_glimpse_enqueue+0x3d3/0x4f0 [mdt] [Thu Apr 11 17:29:46 2019][2873906.979350] [] mdt_intent_glimpse+0x1f/0x30 [mdt] [Thu Apr 11 17:29:46 2019][2873906.985922] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 17:29:46 2019][2873906.992577] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873906.999501] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873907.006781] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873907.013118] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873907.020232] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873907.028129] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:29:46 2019][2873907.034630] [] kthread+0xd1/0xe0 [Thu Apr 11 17:29:46 2019][2873907.039717] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:29:46 2019][2873907.046363] [] 0xffffffffffffffff [Thu Apr 11 17:29:46 2019][2873907.051558] LustreError: dumping log to /tmp/lustre-log.1555028986.118347 [Thu Apr 11 17:29:50 2019][2873911.011774] LNet: Service thread pid 118518 was inactive for 200.62s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 17:29:50 2019][2873911.028969] Pid: 118518, comm: mdt02_085 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:29:50 2019][2873911.038968] Call Trace: [Thu Apr 11 17:29:50 2019][2873911.041614] [] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.048404] [] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.055281] [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.062218] [] mdt_do_glimpse+0x1e9/0x4c0 [mdt] [Thu Apr 11 17:29:50 2019][2873911.068636] [] mdt_glimpse_enqueue+0x3d3/0x4f0 [mdt] [Thu Apr 11 17:29:50 2019][2873911.075469] [] mdt_intent_glimpse+0x1f/0x30 [mdt] [Thu Apr 11 17:29:50 2019][2873911.082044] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 17:29:50 2019][2873911.088702] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.095641] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.102926] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.109279] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.116393] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.124309] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:29:50 2019][2873911.130825] [] kthread+0xd1/0xe0 [Thu Apr 11 17:29:50 2019][2873911.135911] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:29:50 2019][2873911.142576] [] 0xffffffffffffffff [Thu Apr 11 17:29:50 2019][2873911.147770] LustreError: dumping log to /tmp/lustre-log.1555028990.118518 [Thu Apr 11 17:29:51 2019][2873912.547804] LNet: Service thread pid 118359 was inactive for 200.34s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 17:29:51 2019][2873912.565029] Pid: 118359, comm: mdt02_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:29:51 2019][2873912.575032] Call Trace: [Thu Apr 11 17:29:51 2019][2873912.577673] [] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Thu Apr 11 17:29:51 2019][2873912.584458] [] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Thu Apr 11 17:29:51 2019][2873912.591329] [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Thu Apr 11 17:29:52 2019][2873912.598264] [] mdt_do_glimpse+0x1e9/0x4c0 [mdt] [Thu Apr 11 17:29:52 2019][2873912.604682] [] mdt_glimpse_enqueue+0x3d3/0x4f0 [mdt] [Thu Apr 11 17:29:52 2019][2873912.611514] [] mdt_intent_glimpse+0x1f/0x30 [mdt] [Thu Apr 11 17:29:52 2019][2873912.618098] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 17:29:52 2019][2873912.624774] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 17:29:52 2019][2873912.631723] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 17:29:52 2019][2873912.639005] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 17:29:52 2019][2873912.645359] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:29:52 2019][2873912.652467] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:29:52 2019][2873912.660382] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:29:52 2019][2873912.666888] [] kthread+0xd1/0xe0 [Thu Apr 11 17:29:52 2019][2873912.671989] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:29:52 2019][2873912.678641] [] 0xffffffffffffffff [Thu Apr 11 17:29:52 2019][2873912.683865] LustreError: dumping log to /tmp/lustre-log.1555028992.118359 [Thu Apr 11 17:29:53 2019][2873914.083832] Pid: 118245, comm: mdt02_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:29:53 2019][2873914.094031] Call Trace: [Thu Apr 11 17:29:53 2019][2873914.096711] [] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.103553] [] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.110470] [] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.117425] [] mdt_do_glimpse+0x1e9/0x4c0 [mdt] [Thu Apr 11 17:29:53 2019][2873914.123847] [] mdt_glimpse_enqueue+0x3d3/0x4f0 [mdt] [Thu Apr 11 17:29:53 2019][2873914.130686] [] mdt_intent_glimpse+0x1f/0x30 [mdt] [Thu Apr 11 17:29:53 2019][2873914.137261] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 17:29:53 2019][2873914.143913] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.150850] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.158126] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.164475] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.171586] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.179487] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:29:53 2019][2873914.186018] [] kthread+0xd1/0xe0 [Thu Apr 11 17:29:53 2019][2873914.191118] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:29:53 2019][2873914.197768] [] 0xffffffffffffffff [Thu Apr 11 17:29:53 2019][2873914.202975] LustreError: dumping log to /tmp/lustre-log.1555028993.118245 [Thu Apr 11 17:29:58 2019][2873918.622678] Lustre: fir-MDT0002: haven't heard from client 1b61c2a5-63d9-36db-d1df-22db8107d60d (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6475b63800, cur 1555028998 expire 1555028848 last 1555028771 [Thu Apr 11 17:29:58 2019][2873918.644932] LNet: Service thread pid 118518 completed after 208.25s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Thu Apr 11 17:29:58 2019][2873918.644945] Lustre: 118212:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:8s); client may timeout. req@ffff8b6278e14500 x1628636854065072/t0(0) o101->896e598c-c403-5bdd-1ecd-8d1fc627731a@10.8.25.9@o2ib6:20/0 lens 1768/0 e 0 to 0 dl 1555028990 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 17:29:58 2019][2873918.645599] LustreError: 117468:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.8.9@o2ib6: deadline 30:2s ago [Thu Apr 11 17:29:58 2019][2873918.645599] req@ffff8b6339a46000 x1629299545066096/t0(0) o101->66cf7df1-2d4d-7a36-2517-82fd97348afd@10.8.8.9@o2ib6:26/0 lens 584/0 e 0 to 0 dl 1555028996 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 17:29:58 2019][2873918.645605] LustreError: 117468:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 19 previous similar messages [Thu Apr 11 17:29:58 2019][2873918.735046] LNet: Skipped 45 previous similar messages [Thu Apr 11 17:29:58 2019][2873918.853638] LNetError: 117118:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Thu Apr 11 17:32:52 2019][2874093.425933] LustreError: 118659:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b8ccd78e450 x1629298318209904/t0(0) o3->95f309fb-7ae1-b5a5-cb40-881c2341a31d@10.9.106.64@o2ib4:28/0 lens 488/440 e 0 to 0 dl 1555029178 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:32:52 2019][2874093.450415] LustreError: 118659:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 6 previous similar messages [Thu Apr 11 17:32:52 2019][2874093.460211] Lustre: fir-MDT0002: Bulk IO read error with 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4), client will retry: rc -107 [Thu Apr 11 17:34:56 2019][2874217.058033] LustreError: 118940:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b8:0x8237:0x0] [Thu Apr 11 17:37:27 2019][2874368.411104] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 17:37:27 2019][2874368.424523] Lustre: Skipped 4 previous similar messages [Thu Apr 11 17:38:11 2019][2874412.147148] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.12@o2ib6) [Thu Apr 11 17:38:11 2019][2874412.154476] Lustre: Skipped 4953 previous similar messages [Thu Apr 11 17:38:27 2019][2874427.861710] Lustre: fir-MDT0002: Client 94d4bb8d-4025-4bb2-3e79-daa06ff1054f (at 10.9.108.15@o2ib4) reconnecting [Thu Apr 11 17:38:27 2019][2874427.872061] Lustre: Skipped 4395 previous similar messages [Thu Apr 11 17:40:15 2019][2874536.435244] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 17:40:31 2019][2874551.613748] LustreError: 119072:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001291c:0x9c8:0x0] [Thu Apr 11 17:43:25 2019][2874725.858555] LustreError: 118729:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b4efdf16c50 x1628870384274720/t0(0) o3->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:24/0 lens 488/440 e 0 to 0 dl 1555029834 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:43:25 2019][2874725.883268] LustreError: 118729:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 15 previous similar messages [Thu Apr 11 17:43:25 2019][2874725.893036] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 17:43:25 2019][2874725.906167] Lustre: Skipped 1 previous similar message [Thu Apr 11 17:44:05 2019][2874765.636474] Lustre: fir-MDT0000: haven't heard from client ffb79534-0892-6e64-3530-4b750739fbd9 (at 10.9.112.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b559de32000, cur 1555029845 expire 1555029695 last 1555029618 [Thu Apr 11 17:44:25 2019][2874786.484699] Lustre: fir-MDT0002: Bulk IO read error with 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4), client will retry: rc -107 [Thu Apr 11 17:44:41 2019][2874801.746118] Lustre: 118374:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:44:41 2019][2874801.746118] req@ffff8b6f3337b900 x1628592102064928/t0(0) o101->fdfab198-010f-1b84-313f-f38aa0c66607@10.8.17.12@o2ib6:16/0 lens 576/3264 e 0 to 0 dl 1555029886 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:44:41 2019][2874801.775694] Lustre: 118374:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1869 previous similar messages [Thu Apr 11 17:45:21 2019][2874841.637970] Lustre: fir-MDT0000: haven't heard from client c2a8576f-a69b-270d-0441-cffe140563a7 (at 10.8.9.6@o2ib6) in 204 seconds. I think it's dead, and I am evicting it. exp ffff8b4cc3936400, cur 1555029921 expire 1555029771 last 1555029717 [Thu Apr 11 17:45:21 2019][2874841.659965] Lustre: Skipped 4 previous similar messages [Thu Apr 11 17:45:44 2019][2874864.639891] Lustre: MGS: haven't heard from client f3d1a57a-f6f2-b055-d584-689ca43c29b1 (at 10.8.9.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5cfbf01400, cur 1555029944 expire 1555029794 last 1555029717 [Thu Apr 11 17:45:44 2019][2874864.660922] Lustre: Skipped 2 previous similar messages [Thu Apr 11 17:45:46 2019][2874867.487466] LustreError: 118284:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555029856, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b70fda05340/0x53d0f21b7dfe57c0 lrc: 3/1,0 mode: --/PR res: [0x2c001308b:0x2d00:0x0].0x0 bits 0x13/0x8 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118284 timeout: 0 lvb_type: 0 [Thu Apr 11 17:45:46 2019][2874867.527303] LustreError: 118284:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 9 previous similar messages [Thu Apr 11 17:45:52 2019][2874872.686123] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 17:45:58 2019][2874879.576719] LustreError: 118215:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555029868, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b6c1ffc8900/0x53d0f21b7e3cfa19 lrc: 3/1,0 mode: --/PR res: [0x2c00107f4:0xceca:0x0].0x0 bits 0x13/0x8 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118215 timeout: 0 lvb_type: 0 [Thu Apr 11 17:45:58 2019][2874879.616462] LustreError: 118215:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 17:46:24 2019][2874905.225912] LNetError: 117130:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 17:47:16 2019][2874956.832484] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 17:47:37 2019][2874978.041735] LNet: Service thread pid 118340 was inactive for 200.55s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 17:47:37 2019][2874978.058934] LNet: Skipped 1 previous similar message [Thu Apr 11 17:47:37 2019][2874978.064091] Pid: 118340, comm: mdt01_077 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:47:37 2019][2874978.074102] Call Trace: [Thu Apr 11 17:47:37 2019][2874978.076744] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 17:47:37 2019][2874978.083867] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 17:47:37 2019][2874978.091235] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 17:47:37 2019][2874978.098159] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 17:47:37 2019][2874978.104812] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 17:47:37 2019][2874978.111036] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 17:47:37 2019][2874978.117780] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 17:47:37 2019][2874978.123658] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:47:37 2019][2874978.130790] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:47:37 2019][2874978.138683] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:47:37 2019][2874978.145194] [] kthread+0xd1/0xe0 [Thu Apr 11 17:47:37 2019][2874978.150281] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:47:37 2019][2874978.156929] [] 0xffffffffffffffff [Thu Apr 11 17:47:37 2019][2874978.162121] LustreError: dumping log to /tmp/lustre-log.1555030057.118340 [Thu Apr 11 17:47:38 2019][2874979.510920] LNet: Service thread pid 118284 was inactive for 202.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 17:47:38 2019][2874979.528145] Pid: 118284, comm: mdt02_032 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:47:38 2019][2874979.538153] Call Trace: [Thu Apr 11 17:47:38 2019][2874979.540801] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 17:47:38 2019][2874979.547921] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 17:47:38 2019][2874979.555300] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 17:47:38 2019][2874979.562302] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 17:47:38 2019][2874979.569501] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 17:47:38 2019][2874979.576170] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 17:47:38 2019][2874979.583357] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 17:47:38 2019][2874979.590127] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 17:47:38 2019][2874979.596797] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 17:47:38 2019][2874979.603730] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 17:47:38 2019][2874979.611013] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 17:47:38 2019][2874979.617417] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:47:39 2019][2874979.624549] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:47:39 2019][2874979.632446] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:47:39 2019][2874979.638971] [] kthread+0xd1/0xe0 [Thu Apr 11 17:47:39 2019][2874979.644060] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:47:39 2019][2874979.650715] [] 0xffffffffffffffff [Thu Apr 11 17:47:49 2019][2874990.329993] LNet: Service thread pid 118309 was inactive for 200.75s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 17:47:49 2019][2874990.347357] Pid: 118309, comm: mdt01_058 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:47:49 2019][2874990.357498] Call Trace: [Thu Apr 11 17:47:49 2019][2874990.360167] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.367319] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.374725] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 17:47:49 2019][2874990.381700] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 17:47:49 2019][2874990.388393] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 17:47:49 2019][2874990.394658] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 17:47:49 2019][2874990.401431] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 17:47:49 2019][2874990.408855] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.416074] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.424046] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.430599] [] kthread+0xd1/0xe0 [Thu Apr 11 17:47:49 2019][2874990.435712] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:47:49 2019][2874990.442366] [] 0xffffffffffffffff [Thu Apr 11 17:47:49 2019][2874990.447646] LustreError: dumping log to /tmp/lustre-log.1555030069.118309 [Thu Apr 11 17:47:49 2019][2874990.490907] Pid: 118215, comm: mdt01_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 17:47:49 2019][2874990.500983] Call Trace: [Thu Apr 11 17:47:49 2019][2874990.503635] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.511142] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.518701] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 17:47:49 2019][2874990.525766] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 17:47:49 2019][2874990.533442] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 17:47:49 2019][2874990.540587] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 17:47:49 2019][2874990.548392] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 17:47:49 2019][2874990.555253] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 17:47:49 2019][2874990.562025] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.569022] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.576372] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.583146] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.590438] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.598377] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 17:47:49 2019][2874990.604966] [] kthread+0xd1/0xe0 [Thu Apr 11 17:47:50 2019][2874990.610750] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 17:47:50 2019][2874990.617621] [] 0xffffffffffffffff [Thu Apr 11 17:47:58 2019][2874998.735499] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 17:47:58 2019][2874998.748896] Lustre: Skipped 2 previous similar messages [Thu Apr 11 17:48:11 2019][2875012.160003] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 17:48:11 2019][2875012.170572] Lustre: Skipped 1920 previous similar messages [Thu Apr 11 17:48:27 2019][2875028.049294] Lustre: fir-MDT0002: Client b7d5a963-1bd8-0aaa-de83-eb1f005c0685 (at 10.9.102.2@o2ib4) reconnecting [Thu Apr 11 17:48:27 2019][2875028.059559] Lustre: Skipped 1926 previous similar messages [Thu Apr 11 17:49:22 2019][2875082.757011] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 17:49:23 2019][2875084.100370] LustreError: 118857:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010808:0xb781:0x0] [Thu Apr 11 17:52:42 2019][2875283.073914] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 17:58:11 2019][2875612.522383] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.19@o2ib6) [Thu Apr 11 17:58:11 2019][2875612.529773] Lustre: Skipped 1977 previous similar messages [Thu Apr 11 17:58:27 2019][2875628.064560] Lustre: fir-MDT0002: Client a132911f-e28d-2e04-1263-47066608ec3c (at 10.8.23.24@o2ib6) reconnecting [Thu Apr 11 17:58:27 2019][2875628.074820] Lustre: Skipped 1965 previous similar messages [Thu Apr 11 17:58:57 2019][2875658.531509] Lustre: 118337:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555030730/real 1555030730] req@ffff8b6c6e27d400 x1630441199907040/t0(0) o104->fir-MDT0001@10.9.106.38@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1555030737 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Thu Apr 11 17:58:57 2019][2875658.559900] Lustre: 118337:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 310 previous similar messages [Thu Apr 11 17:59:05 2019][2875666.181663] Lustre: 118273:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 17:59:05 2019][2875666.181663] req@ffff8b6c1ef86900 x1628594030556608/t0(0) o36->333c7036-d8a7-2a4b-46a1-9d4709835c95@10.8.17.26@o2ib6:10/0 lens 512/2888 e 1 to 0 dl 1555030750 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:59:05 2019][2875666.210737] Lustre: 118273:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages [Thu Apr 11 17:59:10 2019][2875671.105762] Lustre: 118419:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 17:59:10 2019][2875671.105762] req@ffff8b5be04ad700 x1629298209686560/t0(0) o101->177e7b01-2e59-d88f-9d7c-c7900c7d6e9c@10.9.102.52@o2ib4:15/0 lens 576/3264 e 1 to 0 dl 1555030755 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:59:18 2019][2875679.569936] Lustre: 118337:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555030751/real 1555030751] req@ffff8b6c6e27d400 x1630441199907040/t0(0) o104->fir-MDT0001@10.9.106.38@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1555030758 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 17:59:18 2019][2875679.597599] Lustre: 118337:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Thu Apr 11 17:59:22 2019][2875683.214013] Lustre: 118335:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 17:59:22 2019][2875683.214013] req@ffff8b5e4ef42a00 x1628640127586880/t0(0) o101->9f82f0f8-542a-a8dc-98e8-087257a709fe@10.9.107.47@o2ib4:27/0 lens 576/3264 e 0 to 0 dl 1555030767 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:59:46 2019][2875707.116490] Lustre: 118426:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 17:59:46 2019][2875707.116490] req@ffff8b50f3f68000 x1628869544712416/t0(0) o101->c08b5f65-9984-b24b-bd43-b974510cc858@10.9.104.33@o2ib4:21/0 lens 576/3264 e 1 to 0 dl 1555030791 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 17:59:53 2019][2875714.607643] Lustre: 118337:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555030786/real 1555030786] req@ffff8b6c6e27d400 x1630441199907040/t0(0) o104->fir-MDT0001@10.9.106.38@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1555030793 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 17:59:54 2019][2875714.635400] Lustre: 118337:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Thu Apr 11 18:00:24 2019][2875744.874257] Lustre: 118477:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-29), not sending early reply [Thu Apr 11 18:00:24 2019][2875744.874257] req@ffff8b70c0a5f800 x1628640127593936/t0(0) o101->9f82f0f8-542a-a8dc-98e8-087257a709fe@10.9.107.47@o2ib4:29/0 lens 576/3264 e 0 to 0 dl 1555030829 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:00:24 2019][2875744.903687] Lustre: 118477:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Thu Apr 11 18:00:25 2019][2875746.109271] LustreError: 118564:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555030735, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8b52e794bf00/0x53d0f21b935bfd15 lrc: 3/1,0 mode: --/PR res: [0x24000cd10:0x194c1:0x0].0x0 bits 0x13/0x0 rrc: 106 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118564 timeout: 0 lvb_type: 0 [Thu Apr 11 18:00:27 2019][2875748.543315] LustreError: 118187:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555030737, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8b6bbf58dc40/0x53d0f21b9369e8bd lrc: 3/1,0 mode: --/PR res: [0x24000cd10:0x194c1:0x0].0x0 bits 0x13/0x0 rrc: 106 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118187 timeout: 0 lvb_type: 0 [Thu Apr 11 18:00:51 2019][2875771.676775] LustreError: 118462:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555030761, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8b792e1d57c0/0x53d0f21b93eaf80e lrc: 3/1,0 mode: --/PR res: [0x24000cd10:0x194c1:0x0].0x0 bits 0x13/0x0 rrc: 106 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118462 timeout: 0 lvb_type: 0 [Thu Apr 11 18:00:58 2019][2875779.442930] Lustre: 118382:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555030851/real 1555030851] req@ffff8b8216ae9e00 x1630441202962544/t0(0) o106->fir-MDT0001@10.9.106.38@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1555030858 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Thu Apr 11 18:00:58 2019][2875779.470719] Lustre: 118382:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [Thu Apr 11 18:00:59 2019][2875779.996944] LustreError: 118461:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555030769, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8b87b9bff740/0x53d0f21b9415cf08 lrc: 3/1,0 mode: --/PR res: [0x24000cd10:0x194c1:0x0].0x0 bits 0x13/0x0 rrc: 106 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118461 timeout: 0 lvb_type: 0 [Thu Apr 11 18:01:10 2019][2875790.652163] Lustre: 118558:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 18:01:10 2019][2875790.652163] req@ffff8b55e1a2f200 x1628640507768864/t0(0) o101->4603dbb6-dbf8-8605-2d00-92e1ab665945@10.9.107.46@o2ib4:14/0 lens 480/568 e 0 to 0 dl 1555030874 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:01:10 2019][2875790.681446] Lustre: 118558:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages [Thu Apr 11 18:01:10 2019][2875791.160167] LustreError: 118329:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555030780, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0001_UUID lock: ffff8b6a1cf386c0/0x53d0f21b945129ed lrc: 3/1,0 mode: --/PR res: [0x24000cd10:0x194c1:0x0].0x0 bits 0x13/0x0 rrc: 108 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118329 timeout: 0 lvb_type: 0 [Thu Apr 11 18:01:10 2019][2875791.200196] LustreError: 118329:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Thu Apr 11 18:01:24 2019][2875805.649479] LustreError: 118337:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.106.38@o2ib4) failed to reply to blocking AST (req@ffff8b6c6e27d400 x1630441199907040 status 0 rc -110), evict it ns: mdt-fir-MDT0001_UUID lock: ffff8b747da81f80/0x53d0f21b92846ecc lrc: 4/0,0 mode: PR/PR res: [0x24000cd10:0x194c1:0x0].0x0 bits 0x13/0x0 rrc: 108 type: IBT flags: 0x60200400000020 nid: 10.9.106.38@o2ib4 remote: 0x17cb013ff8810e9 expref: 15 pid: 118223 timeout: 2875883 lvb_type: 0 [Thu Apr 11 18:01:25 2019][2875805.693004] LustreError: 138-a: fir-MDT0001: A client on nid 10.9.106.38@o2ib4 was evicted due to a lock blocking callback time out: rc -110 [Thu Apr 11 18:01:25 2019][2875805.705965] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 155s: evicting client at 10.9.106.38@o2ib4 ns: mdt-fir-MDT0001_UUID lock: ffff8b747da81f80/0x53d0f21b92846ecc lrc: 3/0,0 mode: PR/PR res: [0x24000cd10:0x194c1:0x0].0x0 bits 0x13/0x0 rrc: 108 type: IBT flags: 0x60200400000020 nid: 10.9.106.38@o2ib4 remote: 0x17cb013ff8810e9 expref: 16 pid: 118223 timeout: 0 lvb_type: 0 [Thu Apr 11 18:01:25 2019][2875805.744613] Lustre: 118329:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (104:1s); client may timeout. req@ffff8b6383f37800 x1630377507089248/t0(0) o101->d97ae4e8-9de5-364b-c28b-7b409ac9e530@10.8.7.4@o2ib6:0/0 lens 584/536 e 1 to 0 dl 1555030884 ref 1 fl Complete:/0/0 rc 0/0 [Thu Apr 11 18:01:25 2019][2875805.773003] Lustre: 118329:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 688 previous similar messages [Thu Apr 11 18:01:56 2019][2875836.658126] Lustre: fir-MDT0000: haven't heard from client 91e06134-10b6-11f6-3388-c41822700d1e (at 10.9.102.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5584389000, cur 1555030916 expire 1555030766 last 1555030689 [Thu Apr 11 18:01:56 2019][2875836.680392] Lustre: Skipped 1 previous similar message [Thu Apr 11 18:02:02 2019][2875842.658432] Lustre: fir-MDT0003: haven't heard from client 91e06134-10b6-11f6-3388-c41822700d1e (at 10.9.102.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6ad3cae400, cur 1555030922 expire 1555030772 last 1555030695 [Thu Apr 11 18:02:02 2019][2875842.680470] Lustre: Skipped 10 previous similar messages [Thu Apr 11 18:03:17 2019][2875918.181030] LustreError: 119078:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011182:0xf6e1:0x0] [Thu Apr 11 18:03:41 2019][2875942.122742] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:06:52 2019][2876133.139827] LustreError: 119025:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b661e3c6850 x1628870386731648/t0(0) o3->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:22/0 lens 488/440 e 0 to 0 dl 1555031242 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:06:52 2019][2876133.164337] LustreError: 119025:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 21 previous similar messages [Thu Apr 11 18:06:52 2019][2876133.174129] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 18:06:56 2019][2876136.781259] LustreError: 118688:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001293d:0x2650:0x0] [Thu Apr 11 18:08:13 2019][2876213.958368] Lustre: fir-MDT0002: Connection restored to 3f1d1974-7ed8-ffd0-1256-fba8ddd54f53 (at 10.8.9.10@o2ib6) [Thu Apr 11 18:08:13 2019][2876213.969114] Lustre: Skipped 1995 previous similar messages [Thu Apr 11 18:08:16 2019][2876217.171731] LustreError: 119186:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b585ff02a00 x1628870386873600/t0(0) o37->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:16/0 lens 448/440 e 0 to 0 dl 1555031326 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:08:27 2019][2876228.555757] Lustre: fir-MDT0002: Client 931a3723-3247-0feb-2066-0d2fae060ca7 (at 10.9.101.66@o2ib4) reconnecting [Thu Apr 11 18:08:27 2019][2876228.566220] Lustre: Skipped 1995 previous similar messages [Thu Apr 11 18:08:30 2019][2876231.455305] LNetError: 117118:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:08:37 2019][2876238.193858] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 18:08:37 2019][2876238.207294] Lustre: Skipped 5 previous similar messages [Thu Apr 11 18:08:37 2019][2876238.502186] Lustre: 118928:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628870386908512 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b5a5a913450 x1628870386908512/t0(0) o4->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:7/0 lens 616/0 e 0 to 0 dl 1555031347 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 18:10:01 2019][2876322.309627] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 18:11:27 2019][2876408.242431] LustreError: 118999:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010817:0x13c3:0x0] [Thu Apr 11 18:11:31 2019][2876411.697143] LustreError: 119078:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b8:0x8c9f:0x0] [Thu Apr 11 18:12:10 2019][2876450.829332] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:14:55 2019][2876616.012518] LustreError: 118811:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b785febd050 x1628648805732800/t0(0) o4->837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579@10.9.104.22@o2ib4:25/0 lens 488/448 e 0 to 0 dl 1555031725 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:14:55 2019][2876616.037138] LustreError: 118811:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 3 previous similar messages [Thu Apr 11 18:14:55 2019][2876616.046832] Lustre: fir-MDT0002: Bulk IO write error with 837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579 (at 10.9.104.22@o2ib4), client will retry: rc = -107 [Thu Apr 11 18:14:55 2019][2876616.060229] Lustre: Skipped 1 previous similar message [Thu Apr 11 18:15:58 2019][2876679.161378] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 18:16:12 2019][2876693.465977] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:18:04 2019][2876805.020395] Lustre: fir-MDT0002: Bulk IO write error with 837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579 (at 10.9.104.22@o2ib4), client will retry: rc = -107 [Thu Apr 11 18:18:04 2019][2876805.033789] Lustre: Skipped 5 previous similar messages [Thu Apr 11 18:18:13 2019][2876814.391046] Lustre: fir-MDT0002: Connection restored to (at 10.9.107.48@o2ib4) [Thu Apr 11 18:18:13 2019][2876814.398531] Lustre: Skipped 1949 previous similar messages [Thu Apr 11 18:18:28 2019][2876828.711090] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Thu Apr 11 18:18:28 2019][2876828.721182] Lustre: Skipped 1942 previous similar messages [Thu Apr 11 18:20:10 2019][2876931.104302] LustreError: 118983:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5984467450 x1628648806737904/t0(0) o4->837f4ea3-1bc1-dbfe-f7d2-0c161c5c3579@10.9.104.22@o2ib4:10/0 lens 488/448 e 0 to 0 dl 1555032040 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:20:10 2019][2876931.128880] LustreError: 118983:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 8 previous similar messages [Thu Apr 11 18:20:25 2019][2876946.290330] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:21:22 2019][2877002.697305] LustreError: 118701:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011249:0x2bd4:0x0] [Thu Apr 11 18:22:43 2019][2877084.416987] LustreError: 119200:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00111aa:0x2b12:0x0] [Thu Apr 11 18:22:44 2019][2877085.340259] LustreError: 119072:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128bd:0x3ecf:0x0] [Thu Apr 11 18:25:05 2019][2877226.014606] LustreError: 119125:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010485:0x2ebf:0x0] [Thu Apr 11 18:26:28 2019][2877309.218536] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 18:28:06 2019][2877407.206975] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Thu Apr 11 18:28:06 2019][2877407.220066] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Thu Apr 11 18:28:13 2019][2877414.483195] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.19@o2ib4) [Thu Apr 11 18:28:13 2019][2877414.490794] Lustre: Skipped 1966 previous similar messages [Thu Apr 11 18:28:28 2019][2877428.800699] Lustre: fir-MDT0002: Client 70db6507-827a-2e40-1c44-099543d1512c (at 10.9.108.33@o2ib4) reconnecting [Thu Apr 11 18:28:28 2019][2877428.811058] Lustre: Skipped 1948 previous similar messages [Thu Apr 11 18:33:09 2019][2877709.763646] LustreError: 118956:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b770af7b850 x1628640711848656/t0(0) o4->398eea62-82fb-c936-bc1a-b5cbed90bbf5@10.8.8.12@o2ib6:9/0 lens 488/448 e 0 to 0 dl 1555032819 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:33:09 2019][2877709.787742] Lustre: fir-MDT0002: Bulk IO write error with 398eea62-82fb-c936-bc1a-b5cbed90bbf5 (at 10.8.8.12@o2ib6), client will retry: rc = -110 [Thu Apr 11 18:33:09 2019][2877709.800989] Lustre: Skipped 1 previous similar message [Thu Apr 11 18:33:28 2019][2877729.497153] Lustre: 118711:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628870389279008 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b7a6321f850 x1628870389279008/t0(0) o4->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:28/0 lens 488/0 e 0 to 0 dl 1555032838 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 18:33:39 2019][2877740.266436] LustreError: 118425:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5a5a916450 x1629289954744608/t0(0) o4->bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83@10.8.26.21@o2ib6:9/0 lens 488/448 e 0 to 0 dl 1555032849 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:33:39 2019][2877740.290875] LustreError: 118425:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 5 previous similar messages [Thu Apr 11 18:35:55 2019][2877876.160398] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 18:38:13 2019][2878014.551753] Lustre: fir-MDT0002: Connection restored to 47f38780-4ea3-67ac-f0a8-c670520d3c31 (at 10.8.7.17@o2ib6) [Thu Apr 11 18:38:13 2019][2878014.562204] Lustre: Skipped 1960 previous similar messages [Thu Apr 11 18:38:30 2019][2878030.850538] Lustre: fir-MDT0002: Client 3c1359c4-6f8a-28e8-3fe1-a0a63b3ad896 (at 10.9.102.1@o2ib4) reconnecting [Thu Apr 11 18:38:30 2019][2878030.860799] Lustre: Skipped 1952 previous similar messages [Thu Apr 11 18:39:26 2019][2878086.972658] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:43:16 2019][2878317.336306] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 18:43:16 2019][2878317.349746] Lustre: Skipped 1 previous similar message [Thu Apr 11 18:44:19 2019][2878380.184308] LustreError: 119216:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6b01614b00 x1628870390292720/t0(0) o37->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:19/0 lens 448/440 e 0 to 0 dl 1555033489 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:44:19 2019][2878380.208873] LustreError: 119216:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 7 previous similar messages [Thu Apr 11 18:45:39 2019][2878460.024540] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:46:15 2019][2878496.541843] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:47:07 2019][2878548.244603] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 18:47:07 2019][2878548.257724] Lustre: Skipped 1 previous similar message [Thu Apr 11 18:48:13 2019][2878614.692579] Lustre: fir-MDT0002: Connection restored to (at 10.8.28.6@o2ib6) [Thu Apr 11 18:48:13 2019][2878614.699904] Lustre: Skipped 1944 previous similar messages [Thu Apr 11 18:48:30 2019][2878631.110246] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Thu Apr 11 18:48:30 2019][2878631.120508] Lustre: Skipped 1942 previous similar messages [Thu Apr 11 18:51:06 2019][2878787.477781] LNetError: 117120:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:52:11 2019][2878851.723297] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 18:54:07 2019][2878968.304221] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 18:54:07 2019][2878968.317836] Lustre: Skipped 7 previous similar messages [Thu Apr 11 18:56:55 2019][2879136.253393] LustreError: 118958:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5925053850 x1628870391390272/t0(0) o4->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:25/0 lens 488/448 e 0 to 0 dl 1555034245 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 18:56:55 2019][2879136.277951] LustreError: 118958:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 11 previous similar messages [Thu Apr 11 18:58:14 2019][2879215.658833] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.18@o2ib6) [Thu Apr 11 18:58:14 2019][2879215.666154] Lustre: Skipped 1947 previous similar messages [Thu Apr 11 18:58:30 2019][2879231.249769] Lustre: fir-MDT0002: Client a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6) reconnecting [Thu Apr 11 18:58:30 2019][2879231.260025] Lustre: Skipped 1945 previous similar messages [Thu Apr 11 19:00:58 2019][2879378.727975] Lustre: fir-MDT0000: haven't heard from client b9313302-eef8-69c6-4aff-20781f65edd5 (at 10.9.112.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4d11657000, cur 1555034458 expire 1555034308 last 1555034231 [Thu Apr 11 19:00:58 2019][2879378.749870] Lustre: Skipped 2 previous similar messages [Thu Apr 11 19:04:16 2019][2879577.345393] Lustre: fir-MDT0002: Bulk IO read error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc -107 [Thu Apr 11 19:06:47 2019][2879728.184546] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 19:08:15 2019][2879815.975442] Lustre: fir-MDT0002: Connection restored to 83b62b04-91a2-2007-7323-cfca5b989386 (at 10.9.103.42@o2ib4) [Thu Apr 11 19:08:15 2019][2879815.986047] Lustre: Skipped 1943 previous similar messages [Thu Apr 11 19:08:30 2019][2879831.604601] Lustre: fir-MDT0002: Client 6217c1b1-b990-f8ec-dde7-efa4a4eb9474 (at 10.9.108.11@o2ib4) reconnecting [Thu Apr 11 19:08:30 2019][2879831.614953] Lustre: Skipped 1945 previous similar messages [Thu Apr 11 19:11:37 2019][2880018.421154] LustreError: 118757:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b7ca8f45450 x1628870392651232/t0(0) o3->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:13/0 lens 488/440 e 0 to 0 dl 1555035103 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 19:11:37 2019][2880018.445647] LustreError: 118757:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 6 previous similar messages [Thu Apr 11 19:11:58 2019][2880039.356883] Lustre: fir-MDT0002: Bulk IO write error with 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4), client will retry: rc = -107 [Thu Apr 11 19:11:58 2019][2880039.370260] Lustre: Skipped 4 previous similar messages [Thu Apr 11 19:12:05 2019][2880045.745283] Lustre: fir-MDT0002: haven't heard from client 53841f1a-489e-cb8a-629f-2d08041a8fc0 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5a0e65ec00, cur 1555035125 expire 1555034975 last 1555034898 [Thu Apr 11 19:12:05 2019][2880045.767202] Lustre: Skipped 4 previous similar messages [Thu Apr 11 19:12:06 2019][2880046.741148] Lustre: fir-MDT0000: haven't heard from client 53841f1a-489e-cb8a-629f-2d08041a8fc0 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8be9a85800, cur 1555035126 expire 1555034976 last 1555034899 [Thu Apr 11 19:12:19 2019][2880060.502600] Lustre: 118861:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628870392713728 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b5cf8e12850 x1628870392713728/t0(0) o4->86eef2fc-6fc9-9a07-71d2-411730e19074@10.9.102.18@o2ib4:25/0 lens 488/0 e 0 to 0 dl 1555035145 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 19:18:15 2019][2880416.065257] Lustre: fir-MDT0002: Connection restored to 71f6346b-6c43-76cc-ee66-27a0e00ec0ce (at 10.9.103.8@o2ib4) [Thu Apr 11 19:18:15 2019][2880416.075783] Lustre: Skipped 1950 previous similar messages [Thu Apr 11 19:18:31 2019][2880432.374759] Lustre: fir-MDT0002: Client 70db6507-827a-2e40-1c44-099543d1512c (at 10.9.108.33@o2ib4) reconnecting [Thu Apr 11 19:18:31 2019][2880432.385113] Lustre: Skipped 1945 previous similar messages [Thu Apr 11 19:28:15 2019][2881016.212786] Lustre: fir-MDT0002: Connection restored to f7850cdc-3473-bc74-2a28-406fa0617404 (at 10.9.108.67@o2ib4) [Thu Apr 11 19:28:15 2019][2881016.223397] Lustre: Skipped 1950 previous similar messages [Thu Apr 11 19:28:32 2019][2881033.518483] Lustre: fir-MDT0002: Client 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6) reconnecting [Thu Apr 11 19:28:32 2019][2881033.528760] Lustre: Skipped 1949 previous similar messages [Thu Apr 11 19:38:15 2019][2881616.335964] Lustre: fir-MDT0002: Connection restored to (at 10.9.107.23@o2ib4) [Thu Apr 11 19:38:15 2019][2881616.343454] Lustre: Skipped 1943 previous similar messages [Thu Apr 11 19:38:33 2019][2881633.995987] Lustre: fir-MDT0002: Client e18a93c6-a11b-0b52-af65-1483de6beb82 (at 10.8.8.18@o2ib6) reconnecting [Thu Apr 11 19:38:33 2019][2881634.006170] Lustre: Skipped 1942 previous similar messages [Thu Apr 11 19:48:15 2019][2882216.662005] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Thu Apr 11 19:48:15 2019][2882216.672532] Lustre: Skipped 1948 previous similar messages [Thu Apr 11 19:48:33 2019][2882234.201317] Lustre: fir-MDT0002: Client 10601291-a183-8deb-41a7-af6620223334 (at 10.9.108.36@o2ib4) reconnecting [Thu Apr 11 19:48:33 2019][2882234.211693] Lustre: Skipped 1945 previous similar messages [Thu Apr 11 19:57:50 2019][2882791.183025] LustreError: 117507:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b551cef7050 x1630377485452624/t0(0) o4->e4292209-c55b-cc00-a078-2d504f5a17aa@10.8.1.24@o2ib6:26/0 lens 488/448 e 0 to 0 dl 1555037876 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 19:57:50 2019][2882791.207435] LustreError: 117507:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 2 previous similar messages [Thu Apr 11 19:57:50 2019][2882791.217120] Lustre: fir-MDT0002: Bulk IO write error with e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6), client will retry: rc = -107 [Thu Apr 11 19:57:50 2019][2882791.230345] Lustre: Skipped 1 previous similar message [Thu Apr 11 19:58:16 2019][2882816.802631] Lustre: fir-MDT0002: Connection restored to bcead966-3991-3be5-a9ca-eba502e01069 (at 10.9.108.31@o2ib4) [Thu Apr 11 19:58:16 2019][2882816.813243] Lustre: Skipped 1942 previous similar messages [Thu Apr 11 19:58:33 2019][2882834.294170] Lustre: fir-MDT0002: Client e4a8aa81-b0b9-5652-c269-3fdf6ff70c2f (at 10.9.102.23@o2ib4) reconnecting [Thu Apr 11 19:58:33 2019][2882834.304520] Lustre: Skipped 1946 previous similar messages [Thu Apr 11 20:02:21 2019][2883061.802003] Lustre: fir-MDT0000: haven't heard from client 5858572d-ad56-4047-39cd-574f6acf93d8 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4d0f4d5000, cur 1555038141 expire 1555037991 last 1555037914 [Thu Apr 11 20:02:21 2019][2883061.823826] Lustre: Skipped 3 previous similar messages [Thu Apr 11 20:08:17 2019][2883418.089398] Lustre: fir-MDT0002: Connection restored to 3f1d1974-7ed8-ffd0-1256-fba8ddd54f53 (at 10.8.9.10@o2ib6) [Thu Apr 11 20:08:17 2019][2883418.099834] Lustre: Skipped 1948 previous similar messages [Thu Apr 11 20:08:33 2019][2883434.328541] Lustre: fir-MDT0002: Client 930cf174-e8e0-277a-21bd-b711955e5992 (at 10.8.1.19@o2ib6) reconnecting [Thu Apr 11 20:08:33 2019][2883434.338720] Lustre: Skipped 1943 previous similar messages [Thu Apr 11 20:18:17 2019][2884018.237146] Lustre: fir-MDT0002: Connection restored to 15562cac-eb22-f844-7b88-546ba9ddee38 (at 10.9.101.71@o2ib4) [Thu Apr 11 20:18:17 2019][2884018.247775] Lustre: Skipped 1947 previous similar messages [Thu Apr 11 20:18:33 2019][2884034.400945] Lustre: fir-MDT0002: Client 35ddd205-c174-6b95-20ef-c38a3193733f (at 10.9.108.35@o2ib4) reconnecting [Thu Apr 11 20:18:33 2019][2884034.411291] Lustre: Skipped 1941 previous similar messages [Thu Apr 11 20:28:17 2019][2884618.419516] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.22@o2ib4) [Thu Apr 11 20:28:17 2019][2884618.427007] Lustre: Skipped 1942 previous similar messages [Thu Apr 11 20:28:33 2019][2884634.502122] Lustre: fir-MDT0002: Client 83b8ff7e-e264-e3fb-e7eb-42bd933886ea (at 10.8.1.23@o2ib6) reconnecting [Thu Apr 11 20:28:33 2019][2884634.512306] Lustre: Skipped 1948 previous similar messages [Thu Apr 11 20:37:08 2019][2885148.844567] Lustre: fir-MDT0000: haven't heard from client 52b5030d-d456-5676-99a7-8ae85ba29e7f (at 10.9.102.39@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4d17e3bc00, cur 1555040228 expire 1555040078 last 1555040001 [Thu Apr 11 20:37:08 2019][2885148.866553] Lustre: Skipped 4 previous similar messages [Thu Apr 11 20:38:17 2019][2885218.554240] Lustre: fir-MDT0002: Connection restored to 3dde01db-8bab-e6a5-6af0-a17c56d94df2 (at 10.9.101.66@o2ib4) [Thu Apr 11 20:38:17 2019][2885218.564872] Lustre: Skipped 1947 previous similar messages [Thu Apr 11 20:38:34 2019][2885235.241143] Lustre: fir-MDT0002: Client 3c1359c4-6f8a-28e8-3fe1-a0a63b3ad896 (at 10.9.102.1@o2ib4) reconnecting [Thu Apr 11 20:38:34 2019][2885235.251410] Lustre: Skipped 1943 previous similar messages [Thu Apr 11 20:43:20 2019][2885521.177624] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 20:48:18 2019][2885818.984500] Lustre: fir-MDT0002: Connection restored to (at 10.8.28.6@o2ib6) [Thu Apr 11 20:48:18 2019][2885818.991822] Lustre: Skipped 1943 previous similar messages [Thu Apr 11 20:48:34 2019][2885835.560649] Lustre: fir-MDT0002: Client 86eef2fc-6fc9-9a07-71d2-411730e19074 (at 10.9.102.18@o2ib4) reconnecting [Thu Apr 11 20:48:34 2019][2885835.571007] Lustre: Skipped 1941 previous similar messages [Thu Apr 11 20:58:18 2019][2886419.185597] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.65@o2ib4) [Thu Apr 11 20:58:18 2019][2886419.193108] Lustre: Skipped 1946 previous similar messages [Thu Apr 11 20:58:34 2019][2886435.698894] Lustre: fir-MDT0002: Client 563d53a1-cc0d-a13f-4381-648a7da03caa (at 10.9.105.60@o2ib4) reconnecting [Thu Apr 11 20:58:34 2019][2886435.709261] Lustre: Skipped 1950 previous similar messages [Thu Apr 11 21:03:23 2019][2886724.648042] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:08:18 2019][2887019.301273] Lustre: fir-MDT0002: Connection restored to (at 10.8.7.13@o2ib6) [Thu Apr 11 21:08:18 2019][2887019.308602] Lustre: Skipped 1947 previous similar messages [Thu Apr 11 21:08:35 2019][2887036.261820] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Thu Apr 11 21:08:35 2019][2887036.271910] Lustre: Skipped 1940 previous similar messages [Thu Apr 11 21:13:41 2019][2887342.507312] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:14:45 2019][2887406.330134] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:15:05 2019][2887426.084129] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 21:15:05 2019][2887426.084129] req@ffff8b5df37a2700 x1628640398313264/t0(0) o101->aaf04a23-fec0-b47e-449c-d8bdcfc72bff@10.8.7.32@o2ib6:10/0 lens 1776/3288 e 1 to 0 dl 1555042510 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 21:15:13 2019][2887434.100374] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 21:15:13 2019][2887434.100374] req@ffff8b6b976d2700 x1628594009837216/t0(0) o101->180d69dd-ef9b-1134-9fe6-1e5c20fe7a82@10.8.22.33@o2ib6:18/0 lens 1768/0 e 1 to 0 dl 1555042518 ref 2 fl New:/0/ffffffff rc 0/-1 [Thu Apr 11 21:15:13 2019][2887434.129532] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 502 previous similar messages [Thu Apr 11 21:15:19 2019][2887440.557351] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 29s: evicting client at 10.8.27.9@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8b6576b16300/0x53d0f21cc59f7757 lrc: 3/0,0 mode: PR/PR res: [0x2c001306c:0x14e77:0x0].0x0 bits 0x13/0x0 rrc: 401 type: IBT flags: 0x60200400000020 nid: 10.8.27.9@o2ib6 remote: 0x1c0c3f70c1697841 expref: 314 pid: 118344 timeout: 2887375 lvb_type: 0 [Thu Apr 11 21:15:19 2019][2887440.597428] Lustre: 118345:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:9s); client may timeout. req@ffff8b6ff9fc1800 x1630383961754016/t0(0) o101->83f54ca4-beb5-b0c8-46fd-04b8cce1d8ed@10.8.29.2@o2ib6:10/0 lens 1768/0 e 1 to 0 dl 1555042510 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 21:15:19 2019][2887440.601856] LustreError: 118264:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.25.18@o2ib6: deadline 20:1s ago [Thu Apr 11 21:15:19 2019][2887440.601856] req@ffff8b79473c9200 x1628592577676896/t0(0) o101->7a475994-4fa5-4f3b-5ff1-5f6e376a0449@10.8.25.18@o2ib6:18/0 lens 1768/0 e 1 to 0 dl 1555042518 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 21:15:19 2019][2887440.601858] LustreError: 118264:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 33 previous similar messages [Thu Apr 11 21:15:19 2019][2887440.629475] LustreError: 118273:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8b74c97abc00 ns: mdt-fir-MDT0002_UUID lock: ffff8b6c826ead00/0x53d0f21cc59fba8d lrc: 3/0,0 mode: PR/PR res: [0x2c001306c:0x14e77:0x0].0x0 bits 0x13/0x0 rrc: 349 type: IBT flags: 0x50200400000020 nid: 10.8.27.9@o2ib6 remote: 0x1c0c3f70c16978a3 expref: 8 pid: 118273 timeout: 0 lvb_type: 0 [Thu Apr 11 21:15:19 2019][2887440.706478] Lustre: 118345:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 273 previous similar messages [Thu Apr 11 21:15:22 2019][2887443.777920] LustreError: 118702:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001120c:0x3b76:0x0] [Thu Apr 11 21:18:18 2019][2887619.394343] Lustre: fir-MDT0002: Connection restored to 8fa6e6c1-7182-9fd7-64d1-4ba8f1c7c75d (at 10.9.103.10@o2ib4) [Thu Apr 11 21:18:18 2019][2887619.404955] Lustre: Skipped 2317 previous similar messages [Thu Apr 11 21:18:35 2019][2887636.729791] Lustre: fir-MDT0002: Client a634883e-678f-acdd-a857-3c8ad28c3204 (at 10.9.106.34@o2ib4) reconnecting [Thu Apr 11 21:18:35 2019][2887636.740175] Lustre: Skipped 2323 previous similar messages [Thu Apr 11 21:28:18 2019][2888219.804127] Lustre: fir-MDT0002: Connection restored to (at 10.9.107.48@o2ib4) [Thu Apr 11 21:28:18 2019][2888219.811628] Lustre: Skipped 1926 previous similar messages [Thu Apr 11 21:28:36 2019][2888236.893224] Lustre: fir-MDT0003: Client bafe047a-89ea-21d5-a24e-5c407648cfc7 (at 10.0.10.3@o2ib7) reconnecting [Thu Apr 11 21:28:36 2019][2888236.903412] Lustre: Skipped 1924 previous similar messages [Thu Apr 11 21:30:00 2019][2888321.228314] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:34:20 2019][2888581.044938] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:38:19 2019][2888820.065616] Lustre: fir-MDT0002: Connection restored to (at 10.9.114.5@o2ib4) [Thu Apr 11 21:38:19 2019][2888820.073025] Lustre: Skipped 1931 previous similar messages [Thu Apr 11 21:38:36 2019][2888837.174186] Lustre: fir-MDT0002: Client e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6) reconnecting [Thu Apr 11 21:38:36 2019][2888837.184369] Lustre: Skipped 1922 previous similar messages [Thu Apr 11 21:41:14 2019][2888995.587381] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:42:19 2019][2889060.100861] LustreError: 119049:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b60db93e850 x1630280117241024/t0(0) o4->6ea7ad88-4c41-3a56-d580-e39b0e85132b@10.8.27.6@o2ib6:25/0 lens 488/448 e 0 to 0 dl 1555044145 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 21:42:19 2019][2889060.125310] Lustre: fir-MDT0002: Bulk IO write error with 6ea7ad88-4c41-3a56-d580-e39b0e85132b (at 10.8.27.6@o2ib6), client will retry: rc = -107 [Thu Apr 11 21:44:05 2019][2889165.921632] LustreError: 119035:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5984461450 x1629297443656960/t0(0) o4->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:4/0 lens 488/448 e 0 to 0 dl 1555044274 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 21:44:05 2019][2889165.945981] Lustre: fir-MDT0002: Bulk IO write error with a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6), client will retry: rc = -107 [Thu Apr 11 21:47:57 2019][2889398.728996] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:48:19 2019][2889420.207378] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.11@o2ib4) [Thu Apr 11 21:48:19 2019][2889420.214872] Lustre: Skipped 1924 previous similar messages [Thu Apr 11 21:48:36 2019][2889437.310273] Lustre: fir-MDT0002: Client 99abaf9a-6d16-bfe7-2d5c-9e199780b8b7 (at 10.9.101.71@o2ib4) reconnecting [Thu Apr 11 21:48:36 2019][2889437.320889] Lustre: Skipped 1925 previous similar messages [Thu Apr 11 21:49:00 2019][2889461.407158] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:52:56 2019][2889697.130608] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 21:53:02 2019][2889703.349075] LustreError: 118711:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001124a:0x36fd:0x0] [Thu Apr 11 21:58:19 2019][2890020.360670] Lustre: fir-MDT0002: Connection restored to bcead966-3991-3be5-a9ca-eba502e01069 (at 10.9.108.31@o2ib4) [Thu Apr 11 21:58:19 2019][2890020.371339] Lustre: Skipped 1920 previous similar messages [Thu Apr 11 21:58:36 2019][2890037.420091] Lustre: fir-MDT0002: Client f5140abb-0b33-a9a1-9459-14bf253ba61e (at 10.9.106.60@o2ib4) reconnecting [Thu Apr 11 21:58:36 2019][2890037.430444] Lustre: Skipped 1924 previous similar messages [Thu Apr 11 21:59:09 2019][2890069.971836] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 22:05:18 2019][2890439.539908] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 22:06:32 2019][2890513.829802] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Thu Apr 11 22:06:32 2019][2890513.842593] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Thu Apr 11 22:08:19 2019][2890620.398685] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.23@o2ib6) [Thu Apr 11 22:08:19 2019][2890620.406011] Lustre: Skipped 1928 previous similar messages [Thu Apr 11 22:08:36 2019][2890637.776163] Lustre: fir-MDT0002: Client 931a3723-3247-0feb-2066-0d2fae060ca7 (at 10.9.101.66@o2ib4) reconnecting [Thu Apr 11 22:08:36 2019][2890637.786524] Lustre: Skipped 1927 previous similar messages [Thu Apr 11 22:10:48 2019][2890769.865258] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 22:13:44 2019][2890945.204569] LustreError: 119117:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012926:0x1455:0x0] [Thu Apr 11 22:18:20 2019][2891221.394974] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.16@o2ib6) [Thu Apr 11 22:18:20 2019][2891221.402387] Lustre: Skipped 1925 previous similar messages [Thu Apr 11 22:18:36 2019][2891237.931384] Lustre: fir-MDT0002: Client f419f2f7-2340-142c-2553-5bd564397f99 (at 10.8.28.6@o2ib6) reconnecting [Thu Apr 11 22:18:37 2019][2891237.941682] Lustre: Skipped 1929 previous similar messages [Thu Apr 11 22:18:39 2019][2891240.327502] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Thu Apr 11 22:18:39 2019][2891240.340436] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Thu Apr 11 22:18:46 2019][2891247.822178] LustreError: 119052:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001120e:0x3d39:0x0] [Thu Apr 11 22:18:49 2019][2891250.924593] LustreError: 119125:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b60db93b050 x1629299385932752/t0(0) o4->39b26b25-3151-384d-34d9-da20c3dc7f78@10.8.7.35@o2ib6:25/0 lens 488/448 e 0 to 0 dl 1555046335 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 22:18:49 2019][2891250.930206] Lustre: fir-MDT0002: Bulk IO write error with 39b26b25-3151-384d-34d9-da20c3dc7f78 (at 10.8.7.35@o2ib6), client will retry: rc = -110 [Thu Apr 11 22:18:50 2019][2891250.963069] LustreError: 119125:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message [Thu Apr 11 22:18:52 2019][2891253.056517] LustreError: 119064:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001077b:0x1708:0x0] [Thu Apr 11 22:20:38 2019][2891359.284242] Lustre: 118702:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629298328911504 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b60db93f450 x1629298328911504/t0(0) o4->95f309fb-7ae1-b5a5-cb40-881c2341a31d@10.9.106.64@o2ib4:13/0 lens 488/0 e 0 to 0 dl 1555046443 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 22:25:41 2019][2891662.643346] LustreError: 118643:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00108bb:0x10f83:0x0] [Thu Apr 11 22:28:20 2019][2891821.581245] Lustre: fir-MDT0002: Connection restored to 1f1457dd-8be9-0520-54b3-f55bdd85996c (at 10.8.7.31@o2ib6) [Thu Apr 11 22:28:20 2019][2891821.591705] Lustre: Skipped 1925 previous similar messages [Thu Apr 11 22:28:37 2019][2891838.053586] Lustre: fir-MDT0002: Client 64821009-3ac8-89b2-b714-0e5d4d7937f9 (at 10.9.101.72@o2ib4) reconnecting [Thu Apr 11 22:28:37 2019][2891838.063947] Lustre: Skipped 1925 previous similar messages [Thu Apr 11 22:28:41 2019][2891842.254205] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 22:28:41 2019][2891842.267065] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Thu Apr 11 22:30:21 2019][2891942.168141] LustreError: 119070:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b61e1f93c50 x1628645057381536/t0(0) o3->cb8178a6-c831-1017-88f1-dffd78663a0f@10.8.1.15@o2ib6:21/0 lens 488/440 e 0 to 0 dl 1555047051 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 22:30:21 2019][2891942.192801] Lustre: fir-MDT0002: Bulk IO read error with cb8178a6-c831-1017-88f1-dffd78663a0f (at 10.8.1.15@o2ib6), client will retry: rc -107 [Thu Apr 11 22:30:21 2019][2891942.205856] Lustre: Skipped 1 previous similar message [Thu Apr 11 22:38:20 2019][2892421.940875] Lustre: fir-MDT0002: Connection restored to (at 10.8.26.21@o2ib6) [Thu Apr 11 22:38:20 2019][2892421.948296] Lustre: Skipped 1929 previous similar messages [Thu Apr 11 22:38:37 2019][2892438.266156] Lustre: fir-MDT0002: Client cb8178a6-c831-1017-88f1-dffd78663a0f (at 10.8.1.15@o2ib6) reconnecting [Thu Apr 11 22:38:37 2019][2892438.276336] Lustre: Skipped 1921 previous similar messages [Thu Apr 11 22:39:11 2019][2892472.099763] LustreError: 118665:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b8ccd78e050 x1629298329739728/t0(0) o4->95f309fb-7ae1-b5a5-cb40-881c2341a31d@10.9.106.64@o2ib4:17/0 lens 488/448 e 0 to 0 dl 1555047557 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 22:39:11 2019][2892472.124351] Lustre: fir-MDT0002: Bulk IO write error with 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4), client will retry: rc = -107 [Thu Apr 11 22:39:11 2019][2892472.137735] Lustre: Skipped 1 previous similar message [Thu Apr 11 22:42:52 2019][2892692.961709] LustreError: 118840:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012970:0x1062:0x0] [Thu Apr 11 22:43:13 2019][2892714.954926] Lustre: 118367:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 22:43:13 2019][2892714.954926] req@ffff8b7a6ca33900 x1628641532800304/t0(0) o101->dc770488-99d1-c399-d5bf-254228b4aa3d@10.8.1.25@o2ib6:18/0 lens 576/3264 e 0 to 0 dl 1555047798 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 22:43:14 2019][2892714.984125] Lustre: 118367:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 271 previous similar messages [Thu Apr 11 22:44:18 2019][2892779.923238] LustreError: 118262:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555047768, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b7676a30fc0/0x53d0f21d7aed8ea2 lrc: 3/1,0 mode: --/PR res: [0x2c0011236:0x1ad7:0x0].0x0 bits 0x13/0x8 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118262 timeout: 0 lvb_type: 0 [Thu Apr 11 22:44:18 2019][2892779.963076] LustreError: 118262:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Thu Apr 11 22:46:09 2019][2892890.210464] LNet: Service thread pid 118262 was inactive for 200.28s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 22:46:09 2019][2892890.227679] LNet: Skipped 1 previous similar message [Thu Apr 11 22:46:09 2019][2892890.232841] Pid: 118262, comm: mdt02_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 22:46:09 2019][2892890.242873] Call Trace: [Thu Apr 11 22:46:09 2019][2892890.245524] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 22:46:09 2019][2892890.252652] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 22:46:09 2019][2892890.260053] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 22:46:09 2019][2892890.267064] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 22:46:09 2019][2892890.274240] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 22:46:09 2019][2892890.280896] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 22:46:09 2019][2892890.288084] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 22:46:09 2019][2892890.294831] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 22:46:09 2019][2892890.301487] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 22:46:09 2019][2892890.311215] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 22:46:09 2019][2892890.318632] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 22:46:09 2019][2892890.324979] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 22:46:09 2019][2892890.332180] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 22:46:09 2019][2892890.340386] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 22:46:09 2019][2892890.347025] [] kthread+0xd1/0xe0 [Thu Apr 11 22:46:09 2019][2892890.352120] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 22:46:09 2019][2892890.358927] [] 0xffffffffffffffff [Thu Apr 11 22:46:09 2019][2892890.364165] LustreError: dumping log to /tmp/lustre-log.1555047969.118262 [Thu Apr 11 22:46:10 2019][2892891.939626] LNet: Service thread pid 118428 was inactive for 202.01s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 22:46:10 2019][2892891.956828] Pid: 118428, comm: mdt01_108 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 22:46:10 2019][2892891.966832] Call Trace: [Thu Apr 11 22:46:10 2019][2892891.969472] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 22:46:10 2019][2892891.976588] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 22:46:10 2019][2892891.983972] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 22:46:11 2019][2892891.990914] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 22:46:11 2019][2892891.997577] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 22:46:11 2019][2892892.003799] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 22:46:11 2019][2892892.010542] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 22:46:11 2019][2892892.016418] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 22:46:11 2019][2892892.023550] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 22:46:11 2019][2892892.031456] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 22:46:11 2019][2892892.037973] [] kthread+0xd1/0xe0 [Thu Apr 11 22:46:11 2019][2892892.043059] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 22:46:11 2019][2892892.049708] [] 0xffffffffffffffff [Thu Apr 11 22:46:56 2019][2892937.028265] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 22:46:56 2019][2892937.041013] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Thu Apr 11 22:48:21 2019][2893022.210962] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Thu Apr 11 22:48:21 2019][2893022.221324] Lustre: Skipped 1943 previous similar messages [Thu Apr 11 22:48:37 2019][2893038.391718] Lustre: fir-MDT0002: Client c1922c67-e67d-59f1-51e4-23d63728d4c0 (at 10.9.103.10@o2ib4) reconnecting [Thu Apr 11 22:48:37 2019][2893038.402081] Lustre: Skipped 1947 previous similar messages [Thu Apr 11 22:50:46 2019][2893167.346296] LustreError: 118665:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b8cfc176050 x1629298975677760/t0(0) o4->40c21896-2b28-d1d6-5611-05f09dc83863@10.9.108.24@o2ib4:22/0 lens 488/448 e 0 to 0 dl 1555048252 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 22:50:46 2019][2893167.370916] Lustre: fir-MDT0002: Bulk IO write error with 40c21896-2b28-d1d6-5611-05f09dc83863 (at 10.9.108.24@o2ib4), client will retry: rc = -107 [Thu Apr 11 22:51:02 2019][2893183.656399] Lustre: 118213:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 22:51:02 2019][2893183.656399] req@ffff8b5a63f87b00 x1630280231455200/t0(0) o101->36907934-2c03-0381-635f-b6c727a32090@10.8.7.24@o2ib6:7/0 lens 576/3264 e 0 to 0 dl 1555048267 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 22:52:07 2019][2893248.604708] LustreError: 118369:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555048237, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b659bfff980/0x53d0f21d8b3a169d lrc: 3/0,1 mode: --/PW res: [0x2c00108dc:0x39b7:0x0].0x0 bits 0x40/0x0 rrc: 14 type: IBT flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 118369 timeout: 0 lvb_type: 0 [Thu Apr 11 22:52:07 2019][2893248.644577] LustreError: 118369:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 22:53:58 2019][2893359.211948] LNet: Service thread pid 118548 was inactive for 200.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 22:53:58 2019][2893359.229175] Pid: 118548, comm: mdt00_096 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 22:53:58 2019][2893359.239177] Call Trace: [Thu Apr 11 22:53:58 2019][2893359.241817] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 22:53:58 2019][2893359.248946] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 22:53:58 2019][2893359.256565] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 22:53:58 2019][2893359.263690] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 22:53:58 2019][2893359.270880] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 22:53:58 2019][2893359.277540] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 22:53:58 2019][2893359.284722] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 22:53:58 2019][2893359.291465] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 22:53:58 2019][2893359.298122] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 22:53:58 2019][2893359.305065] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 22:53:58 2019][2893359.312357] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 22:53:58 2019][2893359.318713] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 22:53:58 2019][2893359.325837] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 22:53:58 2019][2893359.333734] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 22:53:58 2019][2893359.340244] [] kthread+0xd1/0xe0 [Thu Apr 11 22:53:58 2019][2893359.345330] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 22:53:58 2019][2893359.351976] [] 0xffffffffffffffff [Thu Apr 11 22:53:58 2019][2893359.357171] LustreError: dumping log to /tmp/lustre-log.1555048438.118548 [Thu Apr 11 22:53:59 2019][2893360.544998] Pid: 118369, comm: mdt00_064 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 22:53:59 2019][2893360.555023] Call Trace: [Thu Apr 11 22:53:59 2019][2893360.557664] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 22:53:59 2019][2893360.564784] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 22:53:59 2019][2893360.572162] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 22:53:59 2019][2893360.579092] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 22:53:59 2019][2893360.585750] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 22:53:59 2019][2893360.591971] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 22:53:59 2019][2893360.598743] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 22:53:59 2019][2893360.604637] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 22:53:59 2019][2893360.611795] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 22:53:59 2019][2893360.619714] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 22:53:59 2019][2893360.626222] [] kthread+0xd1/0xe0 [Thu Apr 11 22:53:59 2019][2893360.631310] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 22:53:59 2019][2893360.637964] [] 0xffffffffffffffff [Thu Apr 11 22:58:21 2019][2893622.900779] Lustre: fir-MDT0002: Connection restored to 30d4579a-fbb3-061d-f340-d61cdf6fe8c4 (at 10.9.108.33@o2ib4) [Thu Apr 11 22:58:21 2019][2893622.911603] Lustre: Skipped 1996 previous similar messages [Thu Apr 11 22:58:27 2019][2893628.587308] LNetError: 117118:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 22:58:27 2019][2893628.600353] LNetError: 117118:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Thu Apr 11 22:58:37 2019][2893638.569747] Lustre: fir-MDT0002: Client 99b2b05a-80e4-edd2-5271-b136b8e4528d (at 10.9.108.67@o2ib4) reconnecting [Thu Apr 11 22:58:37 2019][2893638.580101] Lustre: Skipped 1994 previous similar messages [Thu Apr 11 23:04:13 2019][2893974.970733] LustreError: 118750:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b67a78ba850 x1630280125894528/t0(0) o4->6ea7ad88-4c41-3a56-d580-e39b0e85132b@10.8.27.6@o2ib6:13/0 lens 488/448 e 0 to 0 dl 1555049083 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 23:04:14 2019][2893974.995214] Lustre: fir-MDT0002: Bulk IO write error with 6ea7ad88-4c41-3a56-d580-e39b0e85132b (at 10.8.27.6@o2ib6), client will retry: rc = -107 [Thu Apr 11 23:08:22 2019][2894223.782055] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.26@o2ib4) [Thu Apr 11 23:08:22 2019][2894223.789550] Lustre: Skipped 2003 previous similar messages [Thu Apr 11 23:08:37 2019][2894238.588938] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Thu Apr 11 23:08:37 2019][2894238.599295] Lustre: Skipped 1998 previous similar messages [Thu Apr 11 23:11:29 2019][2894410.517315] LustreError: 119066:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b628af2d400 x1628719234850096/t0(0) o37->ecfe048c-31b4-368b-947d-642490fbb46b@10.9.105.41@o2ib4:5/0 lens 448/440 e 0 to 0 dl 1555049495 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 23:11:43 2019][2894424.412045] LustreError: 118964:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011166:0x1cc5:0x0] [Thu Apr 11 23:14:18 2019][2894579.043149] LustreError: 118904:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001080e:0x68c1:0x0] [Thu Apr 11 23:14:35 2019][2894596.026604] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 23:14:35 2019][2894596.039389] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Thu Apr 11 23:17:04 2019][2894745.229152] LustreError: 118630:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001121d:0x32c0:0x0] [Thu Apr 11 23:18:23 2019][2894824.161695] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.18@o2ib4) [Thu Apr 11 23:18:23 2019][2894824.169185] Lustre: Skipped 1999 previous similar messages [Thu Apr 11 23:18:37 2019][2894838.737027] Lustre: fir-MDT0002: Client 931a3723-3247-0feb-2066-0d2fae060ca7 (at 10.9.101.66@o2ib4) reconnecting [Thu Apr 11 23:18:37 2019][2894838.747395] Lustre: Skipped 2007 previous similar messages [Thu Apr 11 23:19:42 2019][2894903.511847] LustreError: 119009:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011177:0x2a4f:0x0] [Thu Apr 11 23:19:44 2019][2894905.035461] LustreError: 119005:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b60db938050 x1628641980350864/t0(0) o4->83b8ff7e-e264-e3fb-e7eb-42bd933886ea@10.8.1.23@o2ib6:13/0 lens 488/448 e 0 to 0 dl 1555050013 ref 1 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 23:19:44 2019][2894905.059948] Lustre: fir-MDT0002: Bulk IO write error with 83b8ff7e-e264-e3fb-e7eb-42bd933886ea (at 10.8.1.23@o2ib6), client will retry: rc = -110 [Thu Apr 11 23:23:03 2019][2895104.685903] Lustre: 119009:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628645064447024 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b60db938050 x1628645064447024/t0(0) o4->cb8178a6-c831-1017-88f1-dffd78663a0f@10.8.1.15@o2ib6:3/0 lens 560/0 e 0 to 0 dl 1555050213 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Thu Apr 11 23:23:03 2019][2895104.715347] Lustre: 119009:0:(tgt_handler.c:562:tgt_handle_recovery()) Skipped 1 previous similar message [Thu Apr 11 23:28:23 2019][2895424.196808] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.6@o2ib6) [Thu Apr 11 23:28:23 2019][2895424.204140] Lustre: Skipped 2013 previous similar messages [Thu Apr 11 23:28:37 2019][2895438.839225] Lustre: fir-MDT0002: Client b14dc57b-b6ae-736a-c8c8-b28de0f48811 (at 10.9.104.57@o2ib4) reconnecting [Thu Apr 11 23:28:37 2019][2895438.849595] Lustre: Skipped 2000 previous similar messages [Thu Apr 11 23:28:46 2019][2895447.663740] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 23:28:46 2019][2895447.676292] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages [Thu Apr 11 23:32:56 2019][2895697.548130] LustreError: 118940:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00106b9:0x27ab:0x0] [Thu Apr 11 23:35:10 2019][2895831.402198] LustreError: 118956:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001063b:0x48db:0x0] [Thu Apr 11 23:38:23 2019][2896024.288986] Lustre: fir-MDT0002: Connection restored to (at 10.9.114.5@o2ib4) [Thu Apr 11 23:38:23 2019][2896024.296408] Lustre: Skipped 1999 previous similar messages [Thu Apr 11 23:38:36 2019][2896038.038953] LustreError: 119232:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00111d3:0x525f:0x0] [Thu Apr 11 23:38:37 2019][2896038.980128] Lustre: fir-MDT0002: Client f2fba508-8519-f4df-9f05-48b416bbca5e (at 10.8.10.32@o2ib6) reconnecting [Thu Apr 11 23:38:37 2019][2896038.990411] Lustre: Skipped 2005 previous similar messages [Thu Apr 11 23:38:38 2019][2896039.780824] LustreError: 118964:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010486:0x144b:0x0] [Thu Apr 11 23:38:39 2019][2896040.918786] LustreError: 118432:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00106b9:0x288a:0x0] [Thu Apr 11 23:38:41 2019][2896042.634996] LustreError: 118964:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0013104:0x6cd:0x0] [Thu Apr 11 23:38:48 2019][2896049.293320] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 23:38:48 2019][2896049.570579] LustreError: 118958:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001122a:0x9078:0x0] [Thu Apr 11 23:46:09 2019][2896490.381988] LustreError: 118999:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128a9:0x1725:0x0] [Thu Apr 11 23:48:02 2019][2896603.881297] LustreError: 119004:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00120f7:0x63b:0x0] [Thu Apr 11 23:48:23 2019][2896624.301535] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.34@o2ib4) [Thu Apr 11 23:48:23 2019][2896624.309039] Lustre: Skipped 1998 previous similar messages [Thu Apr 11 23:48:38 2019][2896639.410649] Lustre: fir-MDT0002: Client 3c1359c4-6f8a-28e8-3fe1-a0a63b3ad896 (at 10.9.102.1@o2ib4) reconnecting [Thu Apr 11 23:48:38 2019][2896639.420984] Lustre: Skipped 2001 previous similar messages [Thu Apr 11 23:48:39 2019][2896640.677867] Lustre: 118440:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Thu Apr 11 23:48:39 2019][2896640.677867] req@ffff8b504b5a5400 x1629686236914272/t146450791155(0) o36->2ee44edf-15e8-aae3-f19f-229c528020de@10.9.108.18@o2ib4:14/0 lens 544/2888 e 1 to 0 dl 1555051724 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 23:48:39 2019][2896640.708372] Lustre: 118440:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Thu Apr 11 23:48:49 2019][2896650.698068] Lustre: 118221:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Thu Apr 11 23:48:49 2019][2896650.698068] req@ffff8b5f32fd7200 x1630377083939536/t0(0) o101->777290b7-3360-8b49-8dc3-69b0aa79af92@10.8.17.22@o2ib6:24/0 lens 576/3264 e 0 to 0 dl 1555051734 ref 2 fl Interpret:/0/0 rc 0/0 [Thu Apr 11 23:49:54 2019][2896715.651353] LustreError: 118344:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555051704, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b6be2c9f980/0x53d0f21e066d7bd6 lrc: 3/1,0 mode: --/PR res: [0x2c0011927:0xc347:0x0].0x0 bits 0x13/0x8 rrc: 8 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118344 timeout: 0 lvb_type: 0 [Thu Apr 11 23:49:54 2019][2896715.691213] LustreError: 118344:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Thu Apr 11 23:49:55 2019][2896716.128306] LNetError: 117129:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Thu Apr 11 23:49:55 2019][2896716.141139] LNetError: 117129:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Thu Apr 11 23:51:44 2019][2896826.033524] LNet: Service thread pid 118344 was inactive for 200.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 23:51:44 2019][2896826.050739] LNet: Skipped 1 previous similar message [Thu Apr 11 23:51:44 2019][2896826.055891] Pid: 118344, comm: mdt01_080 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 23:51:44 2019][2896826.065886] Call Trace: [Thu Apr 11 23:51:45 2019][2896826.068528] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 23:51:45 2019][2896826.075645] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 23:51:45 2019][2896826.083021] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Thu Apr 11 23:51:45 2019][2896826.090032] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Thu Apr 11 23:51:45 2019][2896826.097217] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Thu Apr 11 23:51:45 2019][2896826.103873] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Thu Apr 11 23:51:45 2019][2896826.111051] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Thu Apr 11 23:51:45 2019][2896826.117800] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Thu Apr 11 23:51:45 2019][2896826.124482] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Thu Apr 11 23:51:45 2019][2896826.131433] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Thu Apr 11 23:51:45 2019][2896826.138736] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Thu Apr 11 23:51:45 2019][2896826.145091] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 23:51:45 2019][2896826.152267] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 23:51:45 2019][2896826.160204] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 23:51:45 2019][2896826.166775] [] kthread+0xd1/0xe0 [Thu Apr 11 23:51:45 2019][2896826.171864] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 23:51:45 2019][2896826.178533] [] 0xffffffffffffffff [Thu Apr 11 23:51:45 2019][2896826.183740] LustreError: dumping log to /tmp/lustre-log.1555051905.118344 [Thu Apr 11 23:51:46 2019][2896827.620781] LNet: Service thread pid 118153 was inactive for 201.96s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Thu Apr 11 23:51:46 2019][2896827.637978] Pid: 118153, comm: mdt00_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Thu Apr 11 23:51:46 2019][2896827.647976] Call Trace: [Thu Apr 11 23:51:46 2019][2896827.650617] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Thu Apr 11 23:51:46 2019][2896827.657747] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Thu Apr 11 23:51:46 2019][2896827.665122] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Thu Apr 11 23:51:46 2019][2896827.672058] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Thu Apr 11 23:51:46 2019][2896827.678713] [] mdt_reint_rec+0x83/0x210 [mdt] [Thu Apr 11 23:51:46 2019][2896827.684933] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Thu Apr 11 23:51:46 2019][2896827.691668] [] mdt_reint+0x67/0x140 [mdt] [Thu Apr 11 23:51:46 2019][2896827.697547] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Thu Apr 11 23:51:46 2019][2896827.704685] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Thu Apr 11 23:51:46 2019][2896827.712582] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Thu Apr 11 23:51:46 2019][2896827.719090] [] kthread+0xd1/0xe0 [Thu Apr 11 23:51:46 2019][2896827.724178] [] ret_from_fork_nospec_begin+0xe/0x21 [Thu Apr 11 23:51:46 2019][2896827.730824] [] 0xffffffffffffffff [Thu Apr 11 23:52:20 2019][2896861.081227] Lustre: fir-MDT0000: haven't heard from client 6005ff27-25fe-bad1-bbec-2de2fb085c90 (at 10.9.112.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b58d7a5d800, cur 1555051940 expire 1555051790 last 1555051713 [Thu Apr 11 23:52:20 2019][2896861.103250] Lustre: Skipped 4 previous similar messages [Thu Apr 11 23:58:23 2019][2897224.649159] Lustre: fir-MDT0002: Connection restored to 5c593326-bba6-222b-6d96-4cc5430ecf5a (at 10.9.103.43@o2ib4) [Thu Apr 11 23:58:23 2019][2897224.659787] Lustre: Skipped 2054 previous similar messages [Thu Apr 11 23:58:38 2019][2897239.438615] Lustre: fir-MDT0002: Client e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6) reconnecting [Thu Apr 11 23:58:38 2019][2897239.448807] Lustre: Skipped 2048 previous similar messages [Fri Apr 12 00:00:08 2019][2897330.057544] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 00:00:08 2019][2897330.070631] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Fri Apr 12 00:06:23 2019][2897704.530510] LustreError: 118930:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b621df0c050 x1630377501540320/t0(0) o4->e4292209-c55b-cc00-a078-2d504f5a17aa@10.8.1.24@o2ib6:23/0 lens 488/448 e 0 to 0 dl 1555052813 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:06:23 2019][2897704.554938] Lustre: fir-MDT0002: Bulk IO write error with e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6), client will retry: rc = -107 [Fri Apr 12 00:08:25 2019][2897826.190666] Lustre: fir-MDT0002: Connection restored to 3f1d1974-7ed8-ffd0-1256-fba8ddd54f53 (at 10.8.9.10@o2ib6) [Fri Apr 12 00:08:25 2019][2897826.201117] Lustre: Skipped 2048 previous similar messages [Fri Apr 12 00:08:38 2019][2897839.691062] Lustre: fir-MDT0002: Client 6ea7ad88-4c41-3a56-d580-e39b0e85132b (at 10.8.27.6@o2ib6) reconnecting [Fri Apr 12 00:08:38 2019][2897839.701242] Lustre: Skipped 2052 previous similar messages [Fri Apr 12 00:09:32 2019][2897893.614877] LustreError: 118930:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0013101:0x248e:0x0] [Fri Apr 12 00:10:14 2019][2897935.164223] LustreError: 118934:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b6e0eed4450 x1629298979358960/t0(0) o4->40c21896-2b28-d1d6-5611-05f09dc83863@10.9.108.24@o2ib4:14/0 lens 488/448 e 0 to 0 dl 1555053044 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:10:14 2019][2897935.188848] Lustre: fir-MDT0002: Bulk IO write error with 40c21896-2b28-d1d6-5611-05f09dc83863 (at 10.9.108.24@o2ib4), client will retry: rc = -107 [Fri Apr 12 00:17:37 2019][2898378.228568] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 00:17:37 2019][2898378.241406] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages [Fri Apr 12 00:17:45 2019][2898386.481293] LustreError: 119049:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00105cc:0x1332:0x0] [Fri Apr 12 00:18:25 2019][2898426.251776] Lustre: fir-MDT0002: Connection restored to f7850cdc-3473-bc74-2a28-406fa0617404 (at 10.9.108.67@o2ib4) [Fri Apr 12 00:18:25 2019][2898426.262402] Lustre: Skipped 2051 previous similar messages [Fri Apr 12 00:18:38 2019][2898439.705507] Lustre: fir-MDT0002: Client eda11444-0027-aea4-d226-92aced58871b (at 10.9.114.5@o2ib4) reconnecting [Fri Apr 12 00:18:38 2019][2898439.715813] Lustre: Skipped 2051 previous similar messages [Fri Apr 12 00:19:03 2019][2898464.840338] Lustre: 118536:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 00:19:03 2019][2898464.840338] req@ffff8b53e2beef00 x1629297059691952/t0(0) o101->fef61529-69a6-3223-f92a-5e13d97909af@10.8.18.9@o2ib6:8/0 lens 576/3264 e 1 to 0 dl 1555053548 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:19:13 2019][2898474.860539] Lustre: 118172:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 00:19:13 2019][2898474.860539] req@ffff8b54208a8f00 x1629292003703952/t146457355165(0) o36->2e7834df-4858-b7f9-7a1a-d9dc0f89f5e0@10.9.108.58@o2ib4:18/0 lens 512/2888 e 0 to 0 dl 1555053558 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:20:18 2019][2898539.813830] LustreError: 118224:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555053528, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b55f43b2400/0x53d0f21e4477ac29 lrc: 3/1,0 mode: --/PR res: [0x2c0011161:0x8369:0x0].0x0 bits 0x13/0x8 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118224 timeout: 0 lvb_type: 0 [Fri Apr 12 00:20:18 2019][2898539.853584] LustreError: 118224:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Fri Apr 12 00:22:09 2019][2898650.326040] LNet: Service thread pid 118224 was inactive for 200.50s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 00:22:09 2019][2898650.343284] Pid: 118224, comm: mdt00_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 00:22:09 2019][2898650.353303] Call Trace: [Fri Apr 12 00:22:09 2019][2898650.355942] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 00:22:09 2019][2898650.363093] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 00:22:09 2019][2898650.370469] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 00:22:09 2019][2898650.377507] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 00:22:09 2019][2898650.384700] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 00:22:09 2019][2898650.391354] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 00:22:09 2019][2898650.398530] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 00:22:09 2019][2898650.405266] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 00:22:09 2019][2898650.411930] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 00:22:09 2019][2898650.418888] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 00:22:09 2019][2898650.426177] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 00:22:09 2019][2898650.432541] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 00:22:09 2019][2898650.439988] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 00:22:09 2019][2898650.449588] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 00:22:09 2019][2898650.456097] [] kthread+0xd1/0xe0 [Fri Apr 12 00:22:09 2019][2898650.461182] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 00:22:09 2019][2898650.467832] [] 0xffffffffffffffff [Fri Apr 12 00:22:09 2019][2898650.473043] LustreError: dumping log to /tmp/lustre-log.1555053729.118224 [Fri Apr 12 00:22:10 2019][2898651.890542] LNet: Service thread pid 118440 was inactive for 202.07s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 00:22:10 2019][2898651.907765] Pid: 118440, comm: mdt00_080 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 00:22:10 2019][2898651.917777] Call Trace: [Fri Apr 12 00:22:10 2019][2898651.920420] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 00:22:10 2019][2898651.927549] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 00:22:10 2019][2898651.934961] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 00:22:10 2019][2898651.941887] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 00:22:10 2019][2898651.948547] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 00:22:10 2019][2898651.954770] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 00:22:10 2019][2898651.961513] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 00:22:10 2019][2898651.967408] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 00:22:10 2019][2898651.974550] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 00:22:10 2019][2898651.982467] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 00:22:10 2019][2898651.988977] [] kthread+0xd1/0xe0 [Fri Apr 12 00:22:10 2019][2898651.994067] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 00:22:10 2019][2898652.000730] [] 0xffffffffffffffff [Fri Apr 12 00:24:32 2019][2898793.638092] LustreError: 118840:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010965:0x606d:0x0] [Fri Apr 12 00:27:54 2019][2898996.038227] LustreError: 118964:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010806:0x2f92:0x0] [Fri Apr 12 00:27:56 2019][2898997.173582] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Fri Apr 12 00:28:25 2019][2899026.753149] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Fri Apr 12 00:28:25 2019][2899026.760547] Lustre: Skipped 2101 previous similar messages [Fri Apr 12 00:28:38 2019][2899039.902326] Lustre: fir-MDT0002: Client f53d399b-a6b5-36d0-034f-8c8c0efbb65f (at 10.9.106.11@o2ib4) reconnecting [Fri Apr 12 00:28:38 2019][2899039.912675] Lustre: Skipped 2096 previous similar messages [Fri Apr 12 00:34:32 2019][2899393.107061] LustreError: 118965:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010921:0x54c:0x0] [Fri Apr 12 00:37:51 2019][2899593.038604] LustreError: 118798:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001309b:0x1d54:0x0] [Fri Apr 12 00:38:08 2019][2899609.942316] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 00:38:08 2019][2899609.954866] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages [Fri Apr 12 00:38:14 2019][2899615.174916] LustreError: 118879:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b55a9eda850 x1629298080631792/t0(0) o4->f5140abb-0b33-a9a1-9459-14bf253ba61e@10.9.106.60@o2ib4:20/0 lens 488/448 e 0 to 0 dl 1555054700 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:38:14 2019][2899615.201303] Lustre: fir-MDT0002: Bulk IO write error with f5140abb-0b33-a9a1-9459-14bf253ba61e (at 10.9.106.60@o2ib4), client will retry: rc = -107 [Fri Apr 12 00:38:25 2019][2899626.829883] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.24@o2ib6) [Fri Apr 12 00:38:25 2019][2899626.837256] Lustre: Skipped 2101 previous similar messages [Fri Apr 12 00:38:39 2019][2899640.150883] Lustre: fir-MDT0002: Client 74806a48-c1e8-fa00-f93a-73121ee183ec (at 10.9.103.43@o2ib4) reconnecting [Fri Apr 12 00:38:39 2019][2899640.161259] Lustre: Skipped 2106 previous similar messages [Fri Apr 12 00:41:30 2019][2899811.689328] LustreError: 118929:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001122d:0x159c:0x0] [Fri Apr 12 00:41:45 2019][2899826.840479] LustreError: 119078:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001051d:0xbdc6:0x0] [Fri Apr 12 00:42:19 2019][2899860.141587] Lustre: fir-MDT0000: haven't heard from client 1aa7042b-f70f-dc58-a180-e365d480adf0 (at 10.8.13.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5842777800, cur 1555054939 expire 1555054789 last 1555054712 [Fri Apr 12 00:42:19 2019][2899860.163483] Lustre: Skipped 4 previous similar messages [Fri Apr 12 00:42:34 2019][2899875.142045] Lustre: fir-MDT0002: haven't heard from client 1aa7042b-f70f-dc58-a180-e365d480adf0 (at 10.8.13.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8bcd6fa800, cur 1555054954 expire 1555054804 last 1555054727 [Fri Apr 12 00:42:34 2019][2899875.163922] Lustre: Skipped 3 previous similar messages [Fri Apr 12 00:45:54 2019][2900075.284900] Lustre: 118488:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 00:45:54 2019][2900075.284900] req@ffff8b79217fe600 x1629292305460912/t146463250401(0) o36->8b903db2-c3fd-349f-174f-3f69d9c7500b@10.9.107.28@o2ib4:29/0 lens 504/2888 e 1 to 0 dl 1555055159 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:46:04 2019][2900085.565128] Lustre: 118339:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 00:46:04 2019][2900085.565128] req@ffff8b6ce95c8c00 x1629290195488272/t0(0) o101->9a6d5107-a0dd-89cc-cc60-368ce86bc790@10.8.17.4@o2ib6:9/0 lens 576/3264 e 0 to 0 dl 1555055169 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:46:39 2019][2900120.277777] Lustre: 118329:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 00:46:39 2019][2900120.277777] req@ffff8b622d039500 x1628580895751856/t146463442843(0) o36->deb76fd8-a5ba-10a3-7858-83fa71fe9af4@10.9.105.53@o2ib4:14/0 lens 544/2888 e 1 to 0 dl 1555055204 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:46:49 2019][2900130.301971] Lustre: 118269:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 00:46:49 2019][2900130.301971] req@ffff8b622d039200 x1628550513491552/t0(0) o101->5e0825a9-6246-92fc-3849-a3736726951c@10.8.2.33@o2ib6:24/0 lens 576/3264 e 0 to 0 dl 1555055214 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:47:09 2019][2900150.362354] LustreError: 118149:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555055139, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b4cdd0c98c0/0x53d0f21e7afc128d lrc: 3/1,0 mode: --/PR res: [0x2c0011186:0xf74:0x0].0x0 bits 0x13/0x8 rrc: 10 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118149 timeout: 0 lvb_type: 0 [Fri Apr 12 00:47:09 2019][2900150.402115] LustreError: 118149:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Fri Apr 12 00:47:54 2019][2900195.234167] LustreError: 118353:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555055184, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b69d5f8dc40/0x53d0f21e7c95f4e1 lrc: 3/1,0 mode: --/PR res: [0x2c001055c:0x29cb:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118353 timeout: 0 lvb_type: 0 [Fri Apr 12 00:47:54 2019][2900195.273907] LustreError: 118353:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Fri Apr 12 00:48:26 2019][2900227.272261] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.19@o2ib6) [Fri Apr 12 00:48:26 2019][2900227.279588] Lustre: Skipped 2119 previous similar messages [Fri Apr 12 00:48:39 2019][2900240.415601] Lustre: fir-MDT0002: Client 64821009-3ac8-89b2-b714-0e5d4d7937f9 (at 10.9.101.72@o2ib4) reconnecting [Fri Apr 12 00:48:39 2019][2900240.425956] Lustre: Skipped 2117 previous similar messages [Fri Apr 12 00:48:59 2019][2900261.110411] LNet: Service thread pid 118359 was inactive for 200.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 00:48:59 2019][2900261.127643] Pid: 118359, comm: mdt02_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 00:49:00 2019][2900261.137658] Call Trace: [Fri Apr 12 00:49:00 2019][2900261.140306] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 00:49:00 2019][2900261.147490] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 00:49:00 2019][2900261.154871] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 00:49:00 2019][2900261.161831] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 00:49:00 2019][2900261.168511] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 00:49:00 2019][2900261.174767] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 00:49:00 2019][2900261.181536] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 00:49:00 2019][2900261.187432] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 00:49:00 2019][2900261.194588] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 00:49:00 2019][2900261.202535] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 00:49:00 2019][2900261.209064] [] kthread+0xd1/0xe0 [Fri Apr 12 00:49:00 2019][2900261.214166] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 00:49:00 2019][2900261.220833] [] 0xffffffffffffffff [Fri Apr 12 00:49:00 2019][2900261.226025] LustreError: dumping log to /tmp/lustre-log.1555055340.118359 [Fri Apr 12 00:49:01 2019][2900262.702887] LNet: Service thread pid 118149 was inactive for 202.33s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 00:49:01 2019][2900262.720124] Pid: 118149, comm: mdt01_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 00:49:01 2019][2900262.730151] Call Trace: [Fri Apr 12 00:49:01 2019][2900262.732807] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 00:49:01 2019][2900262.740451] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 00:49:01 2019][2900262.747899] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 00:49:01 2019][2900262.754918] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 00:49:01 2019][2900262.762094] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 00:49:01 2019][2900262.768760] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 00:49:01 2019][2900262.775943] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 00:49:01 2019][2900262.782696] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 00:49:01 2019][2900262.789352] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 00:49:01 2019][2900262.796312] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 00:49:01 2019][2900262.803618] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 00:49:01 2019][2900262.809971] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 00:49:01 2019][2900262.817109] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 00:49:01 2019][2900262.825018] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 00:49:01 2019][2900262.831541] [] kthread+0xd1/0xe0 [Fri Apr 12 00:49:01 2019][2900262.836628] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 00:49:01 2019][2900262.843276] [] 0xffffffffffffffff [Fri Apr 12 00:49:44 2019][2900305.655294] LNet: Service thread pid 118368 was inactive for 200.41s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 00:49:44 2019][2900305.673255] Pid: 118368, comm: mdt01_091 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 00:49:44 2019][2900305.683515] Call Trace: [Fri Apr 12 00:49:44 2019][2900305.686170] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900305.693526] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900305.703137] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 00:49:44 2019][2900305.711074] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 00:49:44 2019][2900305.717816] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 00:49:44 2019][2900305.725399] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 00:49:44 2019][2900305.732213] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 00:49:44 2019][2900305.738263] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900305.748076] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900305.756003] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900305.762561] [] kthread+0xd1/0xe0 [Fri Apr 12 00:49:44 2019][2900305.767718] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 00:49:44 2019][2900305.774429] [] 0xffffffffffffffff [Fri Apr 12 00:49:44 2019][2900305.779634] LustreError: dumping log to /tmp/lustre-log.1555055384.118368 [Fri Apr 12 00:49:44 2019][2900305.940037] Pid: 118353, comm: mdt01_086 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 00:49:44 2019][2900305.950198] Call Trace: [Fri Apr 12 00:49:44 2019][2900305.952867] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900305.959994] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900305.967594] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 00:49:44 2019][2900305.975071] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 00:49:44 2019][2900305.982430] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 00:49:44 2019][2900305.989157] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 00:49:44 2019][2900305.996426] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 00:49:44 2019][2900306.003194] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 00:49:44 2019][2900306.009867] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900306.016875] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900306.024502] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900306.030965] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900306.038260] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900306.046213] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 00:49:44 2019][2900306.052796] [] kthread+0xd1/0xe0 [Fri Apr 12 00:49:44 2019][2900306.057926] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 00:49:44 2019][2900306.064630] [] 0xffffffffffffffff [Fri Apr 12 00:50:06 2019][2900327.585587] Lustre: 118643:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629298336596320 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b5a5a911850 x1629298336596320/t0(0) o4->95f309fb-7ae1-b5a5-cb40-881c2341a31d@10.9.106.64@o2ib4:6/0 lens 1648/0 e 0 to 0 dl 1555055436 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 00:50:42 2019][2900363.828119] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 00:55:30 2019][2900652.032863] LustreError: 119078:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b621df0c050 x1628758167300304/t0(0) o3->91934493-be40-5b23-a558-2943654113ab@10.8.27.16@o2ib6:6/0 lens 488/440 e 0 to 0 dl 1555055736 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 00:55:30 2019][2900652.057231] Lustre: fir-MDT0002: Bulk IO read error with 91934493-be40-5b23-a558-2943654113ab (at 10.8.27.16@o2ib6), client will retry: rc -107 [Fri Apr 12 00:57:42 2019][2900783.452873] LustreError: 119049:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128f9:0x3d15:0x0] [Fri Apr 12 00:58:26 2019][2900827.427054] Lustre: fir-MDT0002: Connection restored to 6f7215ef-bd09-2e06-26c3-57010db182ff (at 10.9.106.5@o2ib4) [Fri Apr 12 00:58:26 2019][2900827.437587] Lustre: Skipped 2197 previous similar messages [Fri Apr 12 00:58:39 2019][2900840.576228] Lustre: fir-MDT0002: Client cb8178a6-c831-1017-88f1-dffd78663a0f (at 10.8.1.15@o2ib6) reconnecting [Fri Apr 12 00:58:39 2019][2900840.586404] Lustre: Skipped 2188 previous similar messages [Fri Apr 12 01:01:50 2019][2901031.550108] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 01:01:50 2019][2901031.563103] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages [Fri Apr 12 01:01:51 2019][2901032.745796] LustreError: 118958:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5a5a916c50 x1629297463273984/t0(0) o4->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:21/0 lens 488/448 e 0 to 0 dl 1555056141 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:01:51 2019][2901032.770236] Lustre: fir-MDT0002: Bulk IO write error with a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6), client will retry: rc = -107 [Fri Apr 12 01:02:39 2019][2901080.646942] LustreError: 119029:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010797:0x47ef:0x0] [Fri Apr 12 01:06:00 2019][2901282.133844] LustreError: 118933:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b551cef7050 x1628758169129584/t0(0) o3->91934493-be40-5b23-a558-2943654113ab@10.8.27.16@o2ib6:0/0 lens 488/440 e 0 to 0 dl 1555056390 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:06:01 2019][2901282.158239] Lustre: fir-MDT0002: Bulk IO read error with 91934493-be40-5b23-a558-2943654113ab (at 10.8.27.16@o2ib6), client will retry: rc -107 [Fri Apr 12 01:07:52 2019][2901393.218615] LustreError: 117513:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001063a:0x5e83:0x0] [Fri Apr 12 01:08:03 2019][2901404.588663] LustreError: 118996:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00111af:0x2a3e:0x0] [Fri Apr 12 01:08:07 2019][2901408.170488] Lustre: fir-MDT0003: haven't heard from client 1ebf24f0-1d89-f749-09d7-c549024bd6f3 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4ce374e000, cur 1555056487 expire 1555056337 last 1555056260 [Fri Apr 12 01:08:27 2019][2901428.329346] Lustre: fir-MDT0002: Connection restored to (at 10.9.102.1@o2ib4) [Fri Apr 12 01:08:27 2019][2901428.336755] Lustre: Skipped 2193 previous similar messages [Fri Apr 12 01:08:39 2019][2901440.671698] Lustre: fir-MDT0002: Client 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4) reconnecting [Fri Apr 12 01:08:39 2019][2901440.682075] Lustre: Skipped 2199 previous similar messages [Fri Apr 12 01:08:51 2019][2901452.831124] LustreError: 119132:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6c1abecb00 x1629297464252656/t0(0) o37->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:21/0 lens 448/440 e 0 to 0 dl 1555056561 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:10:33 2019][2901555.148745] LustreError: 119049:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b60db93c050 x1628758170389952/t0(0) o4->91934493-be40-5b23-a558-2943654113ab@10.8.27.16@o2ib6:3/0 lens 488/448 e 0 to 0 dl 1555056663 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:10:34 2019][2901555.169902] Lustre: fir-MDT0002: Bulk IO write error with 91934493-be40-5b23-a558-2943654113ab (at 10.8.27.16@o2ib6), client will retry: rc = -107 [Fri Apr 12 01:10:34 2019][2901555.186641] LustreError: 119049:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 1 previous similar message [Fri Apr 12 01:12:37 2019][2901678.963100] LustreError: 119029:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011204:0x5417:0x0] [Fri Apr 12 01:12:39 2019][2901681.108306] LustreError: 119005:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001124a:0x4213:0x0] [Fri Apr 12 01:12:41 2019][2901682.665907] LNetError: 117118:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Fri Apr 12 01:12:41 2019][2901682.679164] LNetError: 117118:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 6 previous similar messages [Fri Apr 12 01:12:59 2019][2901700.252830] Lustre: 118517:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:12:59 2019][2901700.252830] req@ffff8b79b21d2400 x1629294410289392/t0(0) o101->5242fb43-250e-32ac-e1fd-ee4be217ce12@10.9.106.47@o2ib4:4/0 lens 576/3264 e 1 to 0 dl 1555056784 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:13:09 2019][2901711.101031] Lustre: 118396:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 01:13:09 2019][2901711.101031] req@ffff8b54208afb00 x1630381212337104/t146469449322(0) o36->3bbd97db-f6db-d236-9db9-ee9fed5363aa@10.9.114.9@o2ib4:14/0 lens 504/2888 e 0 to 0 dl 1555056794 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:14:14 2019][2901775.286208] LustreError: 118532:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555056764, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b781abb8000/0x53d0f21eb55a92d8 lrc: 3/1,0 mode: --/PR res: [0x2c001293a:0xc28:0x0].0x0 bits 0x13/0x8 rrc: 11 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118532 timeout: 0 lvb_type: 0 [Fri Apr 12 01:14:14 2019][2901775.325950] LustreError: 118532:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Fri Apr 12 01:16:04 2019][2901885.716436] LNet: Service thread pid 118178 was inactive for 200.39s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 01:16:04 2019][2901885.733702] LNet: Skipped 1 previous similar message [Fri Apr 12 01:16:04 2019][2901885.738891] Pid: 118178, comm: mdt02_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 01:16:04 2019][2901885.749106] Call Trace: [Fri Apr 12 01:16:04 2019][2901885.751772] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 01:16:04 2019][2901885.759061] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 01:16:04 2019][2901885.768729] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 01:16:04 2019][2901885.775855] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 01:16:04 2019][2901885.783081] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 01:16:04 2019][2901885.789753] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 01:16:04 2019][2901885.798210] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 01:16:04 2019][2901885.806267] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 01:16:04 2019][2901885.814214] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 01:16:04 2019][2901885.821530] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 01:16:04 2019][2901885.829544] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 01:16:04 2019][2901885.837839] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 01:16:04 2019][2901885.845190] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 01:16:04 2019][2901885.853555] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 01:16:04 2019][2901885.860104] [] kthread+0xd1/0xe0 [Fri Apr 12 01:16:04 2019][2901885.865270] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 01:16:04 2019][2901885.871974] [] 0xffffffffffffffff [Fri Apr 12 01:16:04 2019][2901885.877223] LustreError: dumping log to /tmp/lustre-log.1555056964.118178 [Fri Apr 12 01:16:06 2019][2901887.668625] LNet: Service thread pid 118451 was inactive for 202.38s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 01:16:06 2019][2901887.686529] Pid: 118451, comm: mdt00_083 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 01:16:06 2019][2901887.696669] Call Trace: [Fri Apr 12 01:16:06 2019][2901887.699873] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.707141] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.714535] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 01:16:06 2019][2901887.721478] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 01:16:06 2019][2901887.728404] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 01:16:06 2019][2901887.735619] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 01:16:06 2019][2901887.742532] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 01:16:06 2019][2901887.748431] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.755863] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.763980] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.770549] [] kthread+0xd1/0xe0 [Fri Apr 12 01:16:06 2019][2901887.776134] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 01:16:06 2019][2901887.782863] [] 0xffffffffffffffff [Fri Apr 12 01:16:06 2019][2901887.788087] Pid: 118532, comm: mdt02_091 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 01:16:06 2019][2901887.798136] Call Trace: [Fri Apr 12 01:16:06 2019][2901887.800792] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.808944] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.816359] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 01:16:06 2019][2901887.823498] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 01:16:06 2019][2901887.830760] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 01:16:06 2019][2901887.838908] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 01:16:06 2019][2901887.846395] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 01:16:06 2019][2901887.853416] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 01:16:06 2019][2901887.860488] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.870344] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.878459] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.884874] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.892260] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.900601] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 01:16:06 2019][2901887.908581] [] kthread+0xd1/0xe0 [Fri Apr 12 01:16:06 2019][2901887.913734] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 01:16:06 2019][2901887.920619] [] 0xffffffffffffffff [Fri Apr 12 01:16:08 2019][2901889.174521] LustreError: 118688:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012972:0x12ce:0x0] [Fri Apr 12 01:16:11 2019][2901892.338156] LustreError: 119025:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010704:0x7979:0x0] [Fri Apr 12 01:18:27 2019][2902028.799775] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.64@o2ib4) [Fri Apr 12 01:18:27 2019][2902028.807274] Lustre: Skipped 2244 previous similar messages [Fri Apr 12 01:18:39 2019][2902040.900191] Lustre: fir-MDT0002: Client eb030a79-9cda-5e09-c87b-4c55bbd9e3db (at 10.9.102.7@o2ib4) reconnecting [Fri Apr 12 01:18:39 2019][2902040.910455] Lustre: Skipped 2227 previous similar messages [Fri Apr 12 01:19:27 2019][2902089.029170] LustreError: 119009:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001083f:0xee66:0x0] [Fri Apr 12 01:19:34 2019][2902095.314502] LustreError: 118847:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011204:0x5549:0x0] [Fri Apr 12 01:19:40 2019][2902101.289260] Lustre: 118951:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628758172284224 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b7202a7f850 x1628758172284224/t0(0) o4->91934493-be40-5b23-a558-2943654113ab@10.8.27.16@o2ib6:16/0 lens 496/0 e 0 to 0 dl 1555057186 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 01:23:21 2019][2902322.907746] LustreError: 119194:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011282:0x5407:0x0] [Fri Apr 12 01:23:23 2019][2902324.383704] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 01:23:23 2019][2902324.396654] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 11 previous similar messages [Fri Apr 12 01:23:31 2019][2902332.598335] LustreError: 118432:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010713:0x2f83:0x0] [Fri Apr 12 01:23:32 2019][2902334.073166] LustreError: 118750:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00130e8:0x929:0x0] [Fri Apr 12 01:26:01 2019][2902482.624024] LustreError: 118999:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b7a55b08050 x1630273535184448/t0(0) o4->48186cb7-0668-1018-7ab1-1555c19d856e@10.9.107.23@o2ib4:7/0 lens 488/448 e 0 to 0 dl 1555057567 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:26:01 2019][2902482.624428] Lustre: fir-MDT0002: Bulk IO write error with 48186cb7-0668-1018-7ab1-1555c19d856e (at 10.9.107.23@o2ib4), client will retry: rc = -107 [Fri Apr 12 01:26:01 2019][2902482.624430] Lustre: Skipped 1 previous similar message [Fri Apr 12 01:26:01 2019][2902482.667307] LustreError: 118999:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 1 previous similar message [Fri Apr 12 01:28:27 2019][2902628.992910] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Fri Apr 12 01:28:27 2019][2902629.003360] Lustre: Skipped 2250 previous similar messages [Fri Apr 12 01:28:39 2019][2902640.957989] Lustre: fir-MDT0002: Client 398eea62-82fb-c936-bc1a-b5cbed90bbf5 (at 10.8.8.12@o2ib6) reconnecting [Fri Apr 12 01:28:39 2019][2902640.968171] Lustre: Skipped 2250 previous similar messages [Fri Apr 12 01:31:57 2019][2902839.126988] Lustre: 118788:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629297467590736 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b5984465850 x1629297467590736/t0(0) o4->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:27/0 lens 1648/0 e 0 to 0 dl 1555057947 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 01:32:03 2019][2902844.926353] LustreError: 118958:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5a5a914450 x1630273602124352/t0(0) o3->64821009-3ac8-89b2-b714-0e5d4d7937f9@10.9.101.72@o2ib4:9/0 lens 488/440 e 0 to 0 dl 1555057929 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:32:03 2019][2902844.951344] Lustre: fir-MDT0002: Bulk IO read error with 64821009-3ac8-89b2-b714-0e5d4d7937f9 (at 10.9.101.72@o2ib4), client will retry: rc -107 [Fri Apr 12 01:32:06 2019][2902847.926952] Lustre: 118364:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:32:06 2019][2902847.926952] req@ffff8b77d2a13600 x1629922360988928/t0(0) o101->57bbaa46-7831-4e51-7262-0b6ec02915b4@10.8.16.8@o2ib6:11/0 lens 576/3264 e 1 to 0 dl 1555057931 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:32:06 2019][2902847.956840] Lustre: 118364:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Fri Apr 12 01:32:07 2019][2902848.928970] Lustre: 118576:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:32:07 2019][2902848.928970] req@ffff8b7201a0d100 x1630482096583984/t0(0) o101->d9fa22e1-94fc-65a0-6a79-61b427e3f413@10.8.15.4@o2ib6:12/0 lens 576/3264 e 1 to 0 dl 1555057932 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:32:07 2019][2902848.958264] Lustre: 118576:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 33 previous similar messages [Fri Apr 12 01:32:13 2019][2902854.661107] Lustre: 118389:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-2), not sending early reply [Fri Apr 12 01:32:13 2019][2902854.661107] req@ffff8b5df37b3000 x1630381212916880/t0(0) o101->3bbd97db-f6db-d236-9db9-ee9fed5363aa@10.9.114.9@o2ib4:18/0 lens 576/3264 e 1 to 0 dl 1555057938 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:32:13 2019][2902854.690592] Lustre: 118389:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 9 previous similar messages [Fri Apr 12 01:32:16 2019][2902857.341136] Lustre: 118580:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 01:32:16 2019][2902857.341136] req@ffff8b77d2a17b00 x1630449164601120/t0(0) o101->4b15a192-8c10-6520-238f-0538231b18f1@10.8.15.3@o2ib6:21/0 lens 576/3264 e 0 to 0 dl 1555057941 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:32:21 2019][2902862.865056] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.9.114.9@o2ib4 ns: mdt-fir-MDT0002_UUID lock: ffff8b6b467645c0/0x53d0f21edf5116db lrc: 3/0,0 mode: PR/PR res: [0x2c001306c:0x1d831:0x0].0x0 bits 0x13/0x0 rrc: 168 type: IBT flags: 0x60200400000020 nid: 10.9.114.9@o2ib4 remote: 0x7b3abe662a125cdb expref: 7453 pid: 118248 timeout: 2902797 lvb_type: 0 [Fri Apr 12 01:32:22 2019][2902863.357648] Lustre: 118557:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (30:1s); client may timeout. req@ffff8b574dfe1800 x1630449164601264/t0(0) o101->4b15a192-8c10-6520-238f-0538231b18f1@10.8.15.3@o2ib6:21/0 lens 576/536 e 0 to 0 dl 1555057941 ref 1 fl Complete:/0/0 rc 0/0 [Fri Apr 12 01:32:22 2019][2902863.372177] LustreError: 118339:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8b8c59a2b000 ns: mdt-fir-MDT0002_UUID lock: ffff8b572a7c0480/0x53d0f21edf51f075 lrc: 1/0,0 mode: --/PR res: [0x2c001306c:0x1d831:0x0].0x0 bits 0x13/0x0 rrc: 134 type: IBT flags: 0x54a01400000020 nid: 10.9.114.9@o2ib4 remote: 0x7b3abe662a125d44 expref: 957 pid: 118339 timeout: 0 lvb_type: 0 [Fri Apr 12 01:32:22 2019][2902863.372356] LustreError: 118339:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [Fri Apr 12 01:32:22 2019][2902863.431957] Lustre: 118557:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 4 previous similar messages [Fri Apr 12 01:32:23 2019][2902864.938384] LustreError: 119269:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012956:0x1bef:0x0] [Fri Apr 12 01:36:03 2019][2903084.623196] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 01:36:03 2019][2903084.636032] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 8 previous similar messages [Fri Apr 12 01:36:42 2019][2903123.988877] LustreError: 118701:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b785febe050 x1630273602656176/t0(0) o4->64821009-3ac8-89b2-b714-0e5d4d7937f9@10.9.101.72@o2ib4:12/0 lens 488/448 e 0 to 0 dl 1555058232 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:36:42 2019][2903124.013565] Lustre: fir-MDT0002: Bulk IO write error with 64821009-3ac8-89b2-b714-0e5d4d7937f9 (at 10.9.101.72@o2ib4), client will retry: rc = -107 [Fri Apr 12 01:36:42 2019][2903124.026961] Lustre: Skipped 1 previous similar message [Fri Apr 12 01:36:51 2019][2903133.088768] LustreError: 118999:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b6e0eed5050 x1629297468287008/t0(0) o4->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:21/0 lens 488/448 e 0 to 0 dl 1555058241 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:36:51 2019][2903133.113228] Lustre: fir-MDT0002: Bulk IO write error with a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6), client will retry: rc = -107 [Fri Apr 12 01:38:27 2019][2903229.076506] Lustre: fir-MDT0002: Connection restored to (at 10.8.8.12@o2ib6) [Fri Apr 12 01:38:27 2019][2903229.083836] Lustre: Skipped 2259 previous similar messages [Fri Apr 12 01:38:39 2019][2903241.066954] Lustre: fir-MDT0002: Client 67753486-aa02-1106-6851-77990d7e25c7 (at 10.9.107.61@o2ib4) reconnecting [Fri Apr 12 01:38:39 2019][2903241.077314] Lustre: Skipped 2266 previous similar messages [Fri Apr 12 01:39:33 2019][2903295.115118] LustreError: 119052:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b621df0c850 x1628593851107824/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:3/0 lens 488/448 e 0 to 0 dl 1555058403 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:39:33 2019][2903295.139571] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 01:40:40 2019][2903361.481215] LustreError: 119200:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b60db93b050 x1628758174564688/t0(0) o4->91934493-be40-5b23-a558-2943654113ab@10.8.27.16@o2ib6:10/0 lens 488/448 e 0 to 0 dl 1555058470 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:40:40 2019][2903361.485375] Lustre: fir-MDT0002: Bulk IO write error with 91934493-be40-5b23-a558-2943654113ab (at 10.8.27.16@o2ib6), client will retry: rc = -107 [Fri Apr 12 01:40:40 2019][2903361.519237] LustreError: 119200:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 2 previous similar messages [Fri Apr 12 01:42:50 2019][2903491.383422] LustreError: 118750:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011174:0x10156:0x0] [Fri Apr 12 01:43:25 2019][2903526.921356] LustreError: 118956:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010866:0x11b1:0x0] [Fri Apr 12 01:43:32 2019][2903533.538205] LustreError: 118666:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b621df0d850 x1630377510144976/t0(0) o4->e4292209-c55b-cc00-a078-2d504f5a17aa@10.8.1.24@o2ib6:8/0 lens 488/448 e 0 to 0 dl 1555058618 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:43:32 2019][2903533.563916] Lustre: fir-MDT0002: Bulk IO write error with e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6), client will retry: rc = -110 [Fri Apr 12 01:43:32 2019][2903533.577498] Lustre: Skipped 6 previous similar messages [Fri Apr 12 01:43:56 2019][2903558.055837] LustreError: 119204:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b60db93b450 x1630273604783152/t0(0) o4->64821009-3ac8-89b2-b714-0e5d4d7937f9@10.9.101.72@o2ib4:2/0 lens 488/448 e 0 to 0 dl 1555058642 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:43:56 2019][2903558.080613] LustreError: 119204:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 23 previous similar messages [Fri Apr 12 01:43:56 2019][2903558.090380] Lustre: fir-MDT0002: Bulk IO write error with 64821009-3ac8-89b2-b714-0e5d4d7937f9 (at 10.9.101.72@o2ib4), client will retry: rc = -107 [Fri Apr 12 01:45:49 2019][2903671.009996] LustreError: 118027:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b78b7b95700 x1628719253753104/t0(0) o37->ecfe048c-31b4-368b-947d-642490fbb46b@10.9.105.41@o2ib4:19/0 lens 448/440 e 0 to 0 dl 1555058779 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:46:14 2019][2903696.136344] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 01:46:14 2019][2903696.149064] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages [Fri Apr 12 01:48:08 2019][2903809.416428] LustreError: 118930:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001293d:0x3357:0x0] [Fri Apr 12 01:48:27 2019][2903829.164975] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.60@o2ib4) [Fri Apr 12 01:48:28 2019][2903829.172466] Lustre: Skipped 2258 previous similar messages [Fri Apr 12 01:48:40 2019][2903841.581129] Lustre: fir-MDT0002: Client 8b903db2-c3fd-349f-174f-3f69d9c7500b (at 10.9.107.28@o2ib4) reconnecting [Fri Apr 12 01:48:40 2019][2903841.592376] Lustre: Skipped 2245 previous similar messages [Fri Apr 12 01:48:45 2019][2903846.872823] LustreError: 118650:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5049369850 x1630280135724512/t0(0) o4->6ea7ad88-4c41-3a56-d580-e39b0e85132b@10.8.27.6@o2ib6:21/0 lens 488/448 e 0 to 0 dl 1555058931 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:48:45 2019][2903846.897314] LustreError: 118650:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 1 previous similar message [Fri Apr 12 01:48:45 2019][2903846.906949] Lustre: fir-MDT0002: Bulk IO write error with 6ea7ad88-4c41-3a56-d580-e39b0e85132b (at 10.8.27.6@o2ib6), client will retry: rc = -107 [Fri Apr 12 01:49:40 2019][2903901.737582] Lustre: 118213:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:49:40 2019][2903901.737582] req@ffff8b6743a5d400 x1629294131435408/t0(0) o101->722dd176-ae82-424c-b354-d2c96936288d@10.9.105.23@o2ib4:15/0 lens 1800/3288 e 1 to 0 dl 1555058985 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:49:40 2019][2903901.766912] Lustre: 118213:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Fri Apr 12 01:49:41 2019][2903902.633651] Lustre: 117552:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:49:41 2019][2903902.633651] req@ffff8b6ae7af3000 x1630376578082640/t0(0) o101->4f687388-033e-c261-8070-2adedfbb0ff4@10.9.106.9@o2ib4:16/0 lens 576/0 e 1 to 0 dl 1555058986 ref 2 fl New:/0/ffffffff rc 0/-1 [Fri Apr 12 01:49:42 2019][2903903.635710] Lustre: 117552:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:49:42 2019][2903903.635710] req@ffff8b61cacb5100 x1629018271915280/t0(0) o101->ae22c01c-5307-0e7f-d2e3-ee46df32a51b@10.9.106.70@o2ib4:17/0 lens 1768/0 e 1 to 0 dl 1555058987 ref 2 fl New:/0/ffffffff rc 0/-1 [Fri Apr 12 01:49:42 2019][2903903.664958] Lustre: 117552:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages [Fri Apr 12 01:49:44 2019][2903905.639853] Lustre: 117552:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:49:44 2019][2903905.639853] req@ffff8b65266dc800 x1629293342579776/t0(0) o101->c6802582-b94d-a600-4745-181267927202@10.9.105.48@o2ib4:19/0 lens 1768/0 e 1 to 0 dl 1555058989 ref 2 fl New:/0/ffffffff rc 0/-1 [Fri Apr 12 01:49:44 2019][2903905.669102] Lustre: 117552:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 51 previous similar messages [Fri Apr 12 01:49:48 2019][2903909.648770] Lustre: 117552:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:49:48 2019][2903909.648770] req@ffff8b7a3e8ce000 x1628746491719584/t0(0) o101->8a917374-de35-9bec-1a4f-0ac3e47f87a8@10.9.105.66@o2ib4:23/0 lens 1768/0 e 1 to 0 dl 1555058993 ref 2 fl New:/0/ffffffff rc 0/-1 [Fri Apr 12 01:49:48 2019][2903909.678019] Lustre: 117552:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 45 previous similar messages [Fri Apr 12 01:49:55 2019][2903916.884880] LustreError: 117280:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 30s: evicting client at 10.8.27.16@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8b605123aac0/0x53d0f21f08174914 lrc: 3/0,0 mode: PR/PR res: [0x2c001306c:0x1e022:0x0].0x0 bits 0x13/0x0 rrc: 409 type: IBT flags: 0x60200400000020 nid: 10.8.27.16@o2ib6 remote: 0xe31db5d79a1975a9 expref: 13542 pid: 118327 timeout: 2903851 lvb_type: 0 [Fri Apr 12 01:49:56 2019][2903917.664042] Lustre: 118176:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 01:49:56 2019][2903917.664042] req@ffff8b5c4d723300 x1628702273693328/t0(0) o101->b10e3798-4a87-245f-b16f-483a63a1fb95@10.9.102.15@o2ib4:1/0 lens 1768/0 e 1 to 0 dl 1555059001 ref 2 fl New:/0/ffffffff rc 0/-1 [Fri Apr 12 01:49:56 2019][2903917.693203] Lustre: 118176:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1143 previous similar messages [Fri Apr 12 01:49:56 2019][2903918.094425] Lustre: 118276:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:10s); client may timeout. req@ffff8b6ae7af3000 x1630376578082640/t0(0) o101->4f687388-033e-c261-8070-2adedfbb0ff4@10.9.106.9@o2ib4:16/0 lens 576/0 e 1 to 0 dl 1555058986 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 01:49:56 2019][2903918.094978] LustreError: 118389:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8b73e4a6c400 ns: mdt-fir-MDT0002_UUID lock: ffff8b6051238d80/0x53d0f21f081758fe lrc: 3/0,0 mode: PR/PR res: [0x2c001306c:0x1e022:0x0].0x0 bits 0x13/0x0 rrc: 399 type: IBT flags: 0x50200400000020 nid: 10.8.27.16@o2ib6 remote: 0xe31db5d79a1975b0 expref: 2447 pid: 118389 timeout: 0 lvb_type: 0 [Fri Apr 12 01:49:56 2019][2903918.094981] LustreError: 118389:0:(ldlm_lockd.c:1357:ldlm_handle_enqueue0()) Skipped 2 previous similar messages [Fri Apr 12 01:49:56 2019][2903918.122121] LustreError: 118327:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b70c0a5e300 x1630441851732080/t0(0) o104->fir-MDT0002@10.8.27.16@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Fri Apr 12 01:49:57 2019][2903918.122124] LustreError: 118327:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message [Fri Apr 12 01:49:57 2019][2903918.133020] LustreError: 118281:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.107.21@o2ib4: deadline 20:1s ago [Fri Apr 12 01:49:57 2019][2903918.133020] req@ffff8b5df86d2700 x1629294613302304/t0(0) o101->ea37ce80-e1ed-f5a8-5b21-946c74653f26@10.9.107.21@o2ib4:25/0 lens 1912/0 e 1 to 0 dl 1555058995 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 01:49:57 2019][2903918.133023] LustreError: 118281:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 1 previous similar message [Fri Apr 12 01:49:57 2019][2903918.245978] Lustre: 118276:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 151 previous similar messages [Fri Apr 12 01:50:03 2019][2903924.714981] LustreError: 119200:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0013093:0x1df5:0x0] [Fri Apr 12 01:50:09 2019][2903930.676571] LustreError: 118350:0:(tgt_handler.c:644:process_req_last_xid()) @@@ Unexpected xid 5c932e40aacc0 vs. last_xid 5c932e40ab5ff [Fri Apr 12 01:50:09 2019][2903930.676571] req@ffff8b69ba757500 x1628595295005888/t0(0) o101->f8537525-661f-4957-2618-b1bb64e86581@10.9.104.64@o2ib4:9/0 lens 376/0 e 0 to 0 dl 1555059039 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [Fri Apr 12 01:50:22 2019][2903944.125367] Lustre: fir-MDT0002: Bulk IO write error with ecfe048c-31b4-368b-947d-642490fbb46b (at 10.9.105.41@o2ib4), client will retry: rc = -107 [Fri Apr 12 01:50:46 2019][2903967.740912] Lustre: 118207:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 01:50:46 2019][2903967.740912] req@ffff8b5d7f3cd700 x1629295026877808/t146478042943(0) o36->5c15ad35-04c4-d6b7-9f40-b4147cbb539c@10.8.1.2@o2ib6:21/0 lens 504/2888 e 0 to 0 dl 1555059051 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:50:46 2019][2903967.770959] Lustre: 118207:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 72 previous similar messages [Fri Apr 12 01:51:51 2019][2904032.361147] LustreError: 118339:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555059021, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b64a8f76300/0x53d0f21f093db555 lrc: 3/1,0 mode: --/PR res: [0x2c001121d:0x39b2:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118339 timeout: 0 lvb_type: 0 [Fri Apr 12 01:51:51 2019][2904032.401175] LustreError: 118339:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Fri Apr 12 01:53:20 2019][2904121.273467] LustreError: 118666:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b697cb35450 x1629293758262560/t0(0) o4->930cf174-e8e0-277a-21bd-b711955e5992@10.8.1.19@o2ib6:20/0 lens 488/448 e 0 to 0 dl 1555059230 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:53:20 2019][2904121.284238] LustreError: 119194:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b6018cef050 x1629293758262592/t0(0) o4->930cf174-e8e0-277a-21bd-b711955e5992@10.8.1.19@o2ib6:20/0 lens 488/448 e 0 to 0 dl 1555059230 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 01:53:20 2019][2904121.284241] LustreError: 119194:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 1 previous similar message [Fri Apr 12 01:53:20 2019][2904121.284255] Lustre: fir-MDT0002: Bulk IO write error with 930cf174-e8e0-277a-21bd-b711955e5992 (at 10.8.1.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 01:53:41 2019][2904142.656335] LNet: Service thread pid 118221 was inactive for 200.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 01:53:41 2019][2904142.673601] LNet: Skipped 1 previous similar message [Fri Apr 12 01:53:41 2019][2904142.678829] Pid: 118221, comm: mdt01_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 01:53:41 2019][2904142.688871] Call Trace: [Fri Apr 12 01:53:41 2019][2904142.691521] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 01:53:41 2019][2904142.698679] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 01:53:41 2019][2904142.706083] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 01:53:41 2019][2904142.713067] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 01:53:41 2019][2904142.719778] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 01:53:41 2019][2904142.726016] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 01:53:41 2019][2904142.732761] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 01:53:41 2019][2904142.738670] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 01:53:41 2019][2904142.745795] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 01:53:41 2019][2904142.753708] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 01:53:41 2019][2904142.760253] [] kthread+0xd1/0xe0 [Fri Apr 12 01:53:41 2019][2904142.765555] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 01:53:41 2019][2904142.772205] [] 0xffffffffffffffff [Fri Apr 12 01:53:41 2019][2904142.777421] LustreError: dumping log to /tmp/lustre-log.1555059221.118221 [Fri Apr 12 01:53:43 2019][2904144.238340] LNet: Service thread pid 118339 was inactive for 201.87s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 01:53:43 2019][2904144.255583] Pid: 118339, comm: mdt01_076 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 01:53:43 2019][2904144.265614] Call Trace: [Fri Apr 12 01:53:43 2019][2904144.268267] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 01:53:43 2019][2904144.275409] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 01:53:43 2019][2904144.282852] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 01:53:43 2019][2904144.291106] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 01:53:43 2019][2904144.298290] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 01:53:43 2019][2904144.304974] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 01:53:43 2019][2904144.313639] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 01:53:43 2019][2904144.320402] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 01:53:43 2019][2904144.327115] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 01:53:43 2019][2904144.334105] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 01:53:43 2019][2904144.341416] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 01:53:43 2019][2904144.348872] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 01:53:43 2019][2904144.356134] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 01:53:43 2019][2904144.364064] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 01:53:43 2019][2904144.370598] [] kthread+0xd1/0xe0 [Fri Apr 12 01:53:43 2019][2904144.376223] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 01:53:43 2019][2904144.382976] [] 0xffffffffffffffff [Fri Apr 12 01:54:37 2019][2904199.160670] LustreError: 119052:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001120c:0x406a:0x0] [Fri Apr 12 01:54:46 2019][2904207.770798] LustreError: 119200:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00111b3:0x1c95e:0x0] [Fri Apr 12 01:55:56 2019][2904277.661849] Lustre: fir-MDT0002: Bulk IO write error with e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6), client will retry: rc = -107 [Fri Apr 12 01:55:56 2019][2904277.675131] Lustre: Skipped 2 previous similar messages [Fri Apr 12 01:56:06 2019][2904287.786595] Lustre: 118650:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1628642192487344 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b57a7375050 x1628642192487344/t0(0) o4->b477f0c2-cfda-70b4-919f-fe195c63921e@10.8.7.22@o2ib6:6/0 lens 488/0 e 0 to 0 dl 1555059396 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 01:57:02 2019][2904343.985257] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 01:57:02 2019][2904343.998060] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages [Fri Apr 12 01:58:28 2019][2904429.278088] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.35@o2ib4) [Fri Apr 12 01:58:28 2019][2904429.285641] Lustre: Skipped 2497 previous similar messages [Fri Apr 12 01:58:41 2019][2904442.333832] Lustre: fir-MDT0002: Client 1fed4b9d-ca5d-165d-887c-2f1234134d0c (at 10.8.9.10@o2ib6) reconnecting [Fri Apr 12 01:58:41 2019][2904442.344040] Lustre: Skipped 2506 previous similar messages [Fri Apr 12 01:59:17 2019][2904478.498254] LustreError: 119232:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001075e:0x234:0x0] [Fri Apr 12 02:07:18 2019][2904959.765853] Lustre: 118965:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1630377513452688 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b621df0b450 x1630377513452688/t0(0) o4->e4292209-c55b-cc00-a078-2d504f5a17aa@10.8.1.24@o2ib6:18/0 lens 896/0 e 0 to 0 dl 1555060068 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 02:07:40 2019][2904981.529790] Lustre: 118606:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629297472298576 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b60db93b050 x1629297472298576/t0(0) o4->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:10/0 lens 512/0 e 0 to 0 dl 1555060090 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 02:08:28 2019][2905029.496707] Lustre: fir-MDT0002: Connection restored to 1f1457dd-8be9-0520-54b3-f55bdd85996c (at 10.8.7.31@o2ib6) [Fri Apr 12 02:08:28 2019][2905029.507148] Lustre: Skipped 2245 previous similar messages [Fri Apr 12 02:08:41 2019][2905042.374405] Lustre: fir-MDT0002: Client e18a93c6-a11b-0b52-af65-1483de6beb82 (at 10.8.8.18@o2ib6) reconnecting [Fri Apr 12 02:08:41 2019][2905042.384644] Lustre: Skipped 2241 previous similar messages [Fri Apr 12 02:08:52 2019][2905053.627027] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 02:08:52 2019][2905053.639916] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages [Fri Apr 12 02:12:44 2019][2905285.439104] LustreError: 118754:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b693fb2b050 x1628594823190496/t0(0) o4->267d8456-47e2-363d-5081-c23100dd5f9b@10.8.27.20@o2ib6:20/0 lens 488/448 e 0 to 0 dl 1555060370 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 02:12:44 2019][2905285.440329] Lustre: fir-MDT0002: Bulk IO write error with 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6), client will retry: rc = -107 [Fri Apr 12 02:12:44 2019][2905285.440331] Lustre: Skipped 4 previous similar messages [Fri Apr 12 02:12:44 2019][2905285.482334] LustreError: 118754:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 9 previous similar messages [Fri Apr 12 02:18:28 2019][2905629.906884] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.53@o2ib4) [Fri Apr 12 02:18:28 2019][2905629.914373] Lustre: Skipped 2244 previous similar messages [Fri Apr 12 02:18:41 2019][2905642.457145] Lustre: fir-MDT0002: Client 267d8456-47e2-363d-5081-c23100dd5f9b (at 10.8.27.20@o2ib6) reconnecting [Fri Apr 12 02:18:41 2019][2905642.467417] Lustre: Skipped 2240 previous similar messages [Fri Apr 12 02:19:12 2019][2905673.442591] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Fri Apr 12 02:19:12 2019][2905673.455429] LNetError: 117125:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Fri Apr 12 02:22:13 2019][2905854.283144] LustreError: 118411:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b5326e93450 x1629299412226704/t0(0) o4->39b26b25-3151-384d-34d9-da20c3dc7f78@10.8.7.35@o2ib6:19/0 lens 488/448 e 0 to 0 dl 1555060939 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 02:22:13 2019][2905854.307712] Lustre: fir-MDT0002: Bulk IO write error with 39b26b25-3151-384d-34d9-da20c3dc7f78 (at 10.8.7.35@o2ib6), client will retry: rc = -110 [Fri Apr 12 02:22:13 2019][2905854.321439] Lustre: Skipped 1 previous similar message [Fri Apr 12 02:24:47 2019][2906009.010133] LustreError: 117513:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00130d3:0x1468:0x0] [Fri Apr 12 02:25:34 2019][2906056.158772] LustreError: 119020:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107f4:0xe004:0x0] [Fri Apr 12 02:28:29 2019][2906230.300035] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.8@o2ib6) [Fri Apr 12 02:28:29 2019][2906230.307349] Lustre: Skipped 2239 previous similar messages [Fri Apr 12 02:28:41 2019][2906242.711779] Lustre: fir-MDT0002: Client fef61529-69a6-3223-f92a-5e13d97909af (at 10.8.18.9@o2ib6) reconnecting [Fri Apr 12 02:28:41 2019][2906242.721999] Lustre: Skipped 2248 previous similar messages [Fri Apr 12 02:29:06 2019][2906267.660696] LustreError: 119094:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b5f32fd0c00 x1629293761044096/t0(0) o37->930cf174-e8e0-277a-21bd-b711955e5992@10.8.1.19@o2ib6:6/0 lens 448/440 e 0 to 0 dl 1555061376 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 02:30:22 2019][2906344.156802] LustreError: 118857:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00105c6:0x29ae:0x0] [Fri Apr 12 02:30:25 2019][2906347.084918] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Fri Apr 12 02:30:25 2019][2906347.098019] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Fri Apr 12 02:32:39 2019][2906481.149385] LustreError: 119052:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011282:0x576a:0x0] [Fri Apr 12 02:38:29 2019][2906830.327458] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.34@o2ib4) [Fri Apr 12 02:38:29 2019][2906830.334952] Lustre: Skipped 2249 previous similar messages [Fri Apr 12 02:38:41 2019][2906842.729485] Lustre: fir-MDT0002: Client 2e7834df-4858-b7f9-7a1a-d9dc0f89f5e0 (at 10.9.108.58@o2ib4) reconnecting [Fri Apr 12 02:38:41 2019][2906842.739832] Lustre: Skipped 2242 previous similar messages [Fri Apr 12 02:42:45 2019][2907086.924423] LNetError: 117118:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 02:42:45 2019][2907086.937485] LNetError: 117118:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Fri Apr 12 02:48:29 2019][2907430.491367] Lustre: fir-MDT0002: Connection restored to fe0dfafd-913e-8f07-208e-abc118273e9e (at 10.8.8.8@o2ib6) [Fri Apr 12 02:48:29 2019][2907430.501721] Lustre: Skipped 2242 previous similar messages [Fri Apr 12 02:48:41 2019][2907443.189787] Lustre: fir-MDT0002: Client 2ee44edf-15e8-aae3-f19f-229c528020de (at 10.9.108.18@o2ib4) reconnecting [Fri Apr 12 02:48:41 2019][2907443.200149] Lustre: Skipped 2239 previous similar messages [Fri Apr 12 02:54:23 2019][2907785.029578] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 02:54:23 2019][2907785.042191] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Fri Apr 12 02:58:29 2019][2908030.528082] Lustre: fir-MDT0002: Connection restored to 6e57a6e6-07ae-4b36-8d72-608075772c63 (at 10.9.106.55@o2ib4) [Fri Apr 12 02:58:29 2019][2908030.538700] Lustre: Skipped 2239 previous similar messages [Fri Apr 12 02:58:42 2019][2908043.784707] Lustre: fir-MDT0002: Client 64821009-3ac8-89b2-b714-0e5d4d7937f9 (at 10.9.101.72@o2ib4) reconnecting [Fri Apr 12 02:58:42 2019][2908043.795071] Lustre: Skipped 2252 previous similar messages [Fri Apr 12 03:01:24 2019][2908205.467920] LustreError: 118763:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b7ca8f45850 x1629749674488496/t0(0) o4->a132911f-e28d-2e04-1263-47066608ec3c@10.8.23.24@o2ib6:24/0 lens 488/448 e 0 to 0 dl 1555063314 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 03:01:24 2019][2908205.474385] Lustre: 118754:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629749674488480 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b67a78b8850 x1629749674488480/t0(0) o4->a132911f-e28d-2e04-1263-47066608ec3c@10.8.23.24@o2ib6:24/0 lens 9912/0 e 0 to 0 dl 1555063314 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 03:01:24 2019][2908205.522856] Lustre: fir-MDT0002: Bulk IO write error with a132911f-e28d-2e04-1263-47066608ec3c (at 10.8.23.24@o2ib6), client will retry: rc = -110 [Fri Apr 12 03:06:54 2019][2908536.243254] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 03:06:54 2019][2908536.256279] LNetError: 117123:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message [Fri Apr 12 03:07:12 2019][2908553.756870] LustreError: 118869:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b8badc7b050 x1628764007539184/t0(0) o4->b14dc57b-b6ae-736a-c8c8-b28de0f48811@10.9.104.57@o2ib4:12/0 lens 488/448 e 0 to 0 dl 1555063662 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 03:07:12 2019][2908553.781460] Lustre: fir-MDT0002: Bulk IO write error with b14dc57b-b6ae-736a-c8c8-b28de0f48811 (at 10.9.104.57@o2ib4), client will retry: rc = -107 [Fri Apr 12 03:08:29 2019][2908630.540325] Lustre: fir-MDT0002: Connection restored to 9af20acb-4644-66cf-ffdb-64cc6425272a (at 10.9.107.28@o2ib4) [Fri Apr 12 03:08:29 2019][2908630.550941] Lustre: Skipped 2245 previous similar messages [Fri Apr 12 03:08:43 2019][2908644.367334] Lustre: fir-MDT0002: Client c1922c67-e67d-59f1-51e4-23d63728d4c0 (at 10.9.103.10@o2ib4) reconnecting [Fri Apr 12 03:08:43 2019][2908644.377687] Lustre: Skipped 2237 previous similar messages [Fri Apr 12 03:10:49 2019][2908770.865743] LustreError: 118982:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0011192:0x3aed:0x0] [Fri Apr 12 03:13:18 2019][2908919.667750] LustreError: 118702:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001306b:0x3538:0x0] [Fri Apr 12 03:18:29 2019][2909230.821521] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.17@o2ib6) [Fri Apr 12 03:18:29 2019][2909230.828895] Lustre: Skipped 2241 previous similar messages [Fri Apr 12 03:18:43 2019][2909244.419957] Lustre: fir-MDT0002: Client 10601291-a183-8deb-41a7-af6620223334 (at 10.9.108.36@o2ib4) reconnecting [Fri Apr 12 03:18:43 2019][2909244.430304] Lustre: Skipped 2236 previous similar messages [Fri Apr 12 03:22:03 2019][2909444.684995] LustreError: 118903:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8b7ca8f47850 x1629749676439472/t0(0) o4->a132911f-e28d-2e04-1263-47066608ec3c@10.8.23.24@o2ib6:9/0 lens 488/448 e 0 to 0 dl 1555064529 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 03:22:03 2019][2909444.709211] Lustre: fir-MDT0002: Bulk IO write error with a132911f-e28d-2e04-1263-47066608ec3c (at 10.8.23.24@o2ib6), client will retry: rc = -110 [Fri Apr 12 03:28:30 2019][2909831.650387] Lustre: fir-MDT0002: Connection restored to 88ba37a5-db4c-fc43-b4b3-806ad9c864f1 (at 10.9.102.2@o2ib4) [Fri Apr 12 03:28:30 2019][2909831.660912] Lustre: Skipped 2234 previous similar messages [Fri Apr 12 03:28:43 2019][2909844.615994] Lustre: fir-MDT0002: Client 40c21896-2b28-d1d6-5611-05f09dc83863 (at 10.9.108.24@o2ib4) reconnecting [Fri Apr 12 03:28:43 2019][2909844.626353] Lustre: Skipped 2250 previous similar messages [Fri Apr 12 03:38:30 2019][2910431.882606] Lustre: fir-MDT0002: Connection restored to (at 10.8.23.24@o2ib6) [Fri Apr 12 03:38:30 2019][2910431.890023] Lustre: Skipped 2253 previous similar messages [Fri Apr 12 03:38:43 2019][2910445.022900] Lustre: fir-MDT0002: Client b14dc57b-b6ae-736a-c8c8-b28de0f48811 (at 10.9.104.57@o2ib4) reconnecting [Fri Apr 12 03:38:43 2019][2910445.033247] Lustre: Skipped 2240 previous similar messages [Fri Apr 12 03:48:30 2019][2911031.904107] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.34@o2ib4) [Fri Apr 12 03:48:30 2019][2911031.911605] Lustre: Skipped 2240 previous similar messages [Fri Apr 12 03:48:43 2019][2911045.180337] Lustre: fir-MDT0002: Client 44aac05d-9ebd-7542-b720-92fdacafd9a2 (at 10.9.104.68@o2ib4) reconnecting [Fri Apr 12 03:48:43 2019][2911045.190696] Lustre: Skipped 2240 previous similar messages [Fri Apr 12 03:53:00 2019][2911301.365962] Lustre: fir-MDT0000: haven't heard from client 18458fbe-69f4-8de1-3b2d-f908d47f7856 (at 10.8.22.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4cd9baa800, cur 1555066380 expire 1555066230 last 1555066153 [Fri Apr 12 03:53:00 2019][2911301.387860] Lustre: Skipped 4 previous similar messages [Fri Apr 12 03:53:19 2019][2911320.375262] Lustre: fir-MDT0002: haven't heard from client 18458fbe-69f4-8de1-3b2d-f908d47f7856 (at 10.8.22.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b784cf13000, cur 1555066399 expire 1555066249 last 1555066172 [Fri Apr 12 03:53:19 2019][2911320.397198] Lustre: Skipped 3 previous similar messages [Fri Apr 12 03:54:29 2019][2911391.041600] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 03:54:29 2019][2911391.054277] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages [Fri Apr 12 03:58:30 2019][2911632.039140] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.2@o2ib6) [Fri Apr 12 03:58:30 2019][2911632.046375] Lustre: Skipped 2235 previous similar messages [Fri Apr 12 03:58:44 2019][2911645.664314] Lustre: fir-MDT0002: Client 1fed4b9d-ca5d-165d-887c-2f1234134d0c (at 10.8.9.10@o2ib6) reconnecting [Fri Apr 12 03:58:44 2019][2911645.674496] Lustre: Skipped 2248 previous similar messages [Fri Apr 12 04:01:36 2019][2911817.722640] LustreError: 118601:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0001: BRW to missing obj [0x240012eac:0x7f0:0x0] [Fri Apr 12 04:08:30 2019][2912232.264800] Lustre: fir-MDT0002: Connection restored to b8776385-d179-aafa-3da1-de7b6320d2e0 (at 10.8.18.18@o2ib6) [Fri Apr 12 04:08:30 2019][2912232.275329] Lustre: Skipped 2251 previous similar messages [Fri Apr 12 04:08:44 2019][2912245.685658] Lustre: fir-MDT0002: Client 9a6d5107-a0dd-89cc-cc60-368ce86bc790 (at 10.8.17.4@o2ib6) reconnecting [Fri Apr 12 04:08:44 2019][2912245.695837] Lustre: Skipped 2234 previous similar messages [Fri Apr 12 04:18:31 2019][2912832.401320] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.57@o2ib4) [Fri Apr 12 04:18:31 2019][2912832.408821] Lustre: Skipped 2236 previous similar messages [Fri Apr 12 04:18:44 2019][2912845.771374] Lustre: fir-MDT0002: Client 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4) reconnecting [Fri Apr 12 04:18:44 2019][2912845.781734] Lustre: Skipped 2244 previous similar messages [Fri Apr 12 04:19:12 2019][2912873.409242] Lustre: MGS: haven't heard from client 98de6bfa-d685-ca3f-246d-c6ee197ecb57 (at 10.8.23.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b8978338c00, cur 1555067952 expire 1555067802 last 1555067725 [Fri Apr 12 04:19:24 2019][2912885.395923] Lustre: fir-MDT0000: haven't heard from client 67c32ec1-9885-4261-69c9-b0a56178e76b (at 10.8.23.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b888f210400, cur 1555067964 expire 1555067814 last 1555067737 [Fri Apr 12 04:19:25 2019][2912886.400617] Lustre: fir-MDT0002: haven't heard from client 67c32ec1-9885-4261-69c9-b0a56178e76b (at 10.8.23.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b781f303c00, cur 1555067965 expire 1555067815 last 1555067738 [Fri Apr 12 04:19:25 2019][2912886.422512] Lustre: Skipped 2 previous similar messages [Fri Apr 12 04:28:31 2019][2913432.582621] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.68@o2ib4) [Fri Apr 12 04:28:31 2019][2913432.590110] Lustre: Skipped 2247 previous similar messages [Fri Apr 12 04:28:44 2019][2913445.929623] Lustre: fir-MDT0002: Client 369e48ad-81ca-5881-14c8-c80031c22f3c (at 10.9.104.17@o2ib4) reconnecting [Fri Apr 12 04:28:44 2019][2913445.939977] Lustre: Skipped 2244 previous similar messages [Fri Apr 12 04:38:31 2019][2914032.876516] Lustre: fir-MDT0002: Connection restored to 15562cac-eb22-f844-7b88-546ba9ddee38 (at 10.9.101.71@o2ib4) [Fri Apr 12 04:38:31 2019][2914032.887130] Lustre: Skipped 2243 previous similar messages [Fri Apr 12 04:38:44 2019][2914045.999388] Lustre: fir-MDT0002: Client 398eea62-82fb-c936-bc1a-b5cbed90bbf5 (at 10.8.8.12@o2ib6) reconnecting [Fri Apr 12 04:38:44 2019][2914046.009580] Lustre: Skipped 2239 previous similar messages [Fri Apr 12 04:48:31 2019][2914633.049938] Lustre: fir-MDT0002: Connection restored to (at 10.8.17.4@o2ib6) [Fri Apr 12 04:48:31 2019][2914633.057264] Lustre: Skipped 2244 previous similar messages [Fri Apr 12 04:48:44 2019][2914646.208670] Lustre: fir-MDT0002: Client 931a3723-3247-0feb-2066-0d2fae060ca7 (at 10.9.101.66@o2ib4) reconnecting [Fri Apr 12 04:48:44 2019][2914646.219021] Lustre: Skipped 2248 previous similar messages [Fri Apr 12 04:58:32 2019][2915233.677474] Lustre: fir-MDT0002: Connection restored to (at 10.8.7.13@o2ib6) [Fri Apr 12 04:58:32 2019][2915233.684810] Lustre: Skipped 2246 previous similar messages [Fri Apr 12 04:58:44 2019][2915246.287602] Lustre: fir-MDT0002: Client 74806a48-c1e8-fa00-f93a-73121ee183ec (at 10.9.103.43@o2ib4) reconnecting [Fri Apr 12 04:58:44 2019][2915246.297951] Lustre: Skipped 2236 previous similar messages [Fri Apr 12 05:08:33 2019][2915834.416672] Lustre: fir-MDT0002: Connection restored to 5c593326-bba6-222b-6d96-4cc5430ecf5a (at 10.9.103.43@o2ib4) [Fri Apr 12 05:08:33 2019][2915834.427284] Lustre: Skipped 2242 previous similar messages [Fri Apr 12 05:08:45 2019][2915846.444476] Lustre: fir-MDT0002: Client 1fed4b9d-ca5d-165d-887c-2f1234134d0c (at 10.8.9.10@o2ib6) reconnecting [Fri Apr 12 05:08:45 2019][2915846.454653] Lustre: Skipped 2239 previous similar messages [Fri Apr 12 05:09:22 2019][2915883.456082] Lustre: fir-MDT0000: haven't heard from client 6d053524-32c3-52e1-bd61-4e8e12a872e4 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6c0e40b800, cur 1555070962 expire 1555070812 last 1555070735 [Fri Apr 12 05:18:33 2019][2916434.578351] Lustre: fir-MDT0002: Connection restored to 3f1d1974-7ed8-ffd0-1256-fba8ddd54f53 (at 10.8.9.10@o2ib6) [Fri Apr 12 05:18:33 2019][2916434.588785] Lustre: Skipped 2247 previous similar messages [Fri Apr 12 05:18:45 2019][2916446.791306] Lustre: fir-MDT0002: Client deb76fd8-a5ba-10a3-7858-83fa71fe9af4 (at 10.9.105.53@o2ib4) reconnecting [Fri Apr 12 05:18:45 2019][2916446.801670] Lustre: Skipped 2248 previous similar messages [Fri Apr 12 05:27:34 2019][2916975.484219] Lustre: fir-MDT0001: haven't heard from client 5ef91bce-8115-9590-7cea-13d860deb9bb (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5b4c755c00, cur 1555072054 expire 1555071904 last 1555071827 [Fri Apr 12 05:27:34 2019][2916975.506092] Lustre: Skipped 4 previous similar messages [Fri Apr 12 05:27:45 2019][2916986.478435] Lustre: fir-MDT0000: haven't heard from client 5ef91bce-8115-9590-7cea-13d860deb9bb (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b64b2720c00, cur 1555072065 expire 1555071915 last 1555071838 [Fri Apr 12 05:28:33 2019][2917034.922238] Lustre: fir-MDT0002: Connection restored to (at 10.9.105.53@o2ib4) [Fri Apr 12 05:28:33 2019][2917034.929742] Lustre: Skipped 2247 previous similar messages [Fri Apr 12 05:28:45 2019][2917046.962397] Lustre: fir-MDT0002: Client 39b26b25-3151-384d-34d9-da20c3dc7f78 (at 10.8.7.35@o2ib6) reconnecting [Fri Apr 12 05:28:45 2019][2917046.972593] Lustre: Skipped 2238 previous similar messages [Fri Apr 12 05:31:15 2019][2917196.482640] Lustre: fir-MDT0000: haven't heard from client f5f6ba85-d55c-3558-68dc-e8526225713b (at 10.8.26.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5c8c779800, cur 1555072275 expire 1555072125 last 1555072048 [Fri Apr 12 05:31:15 2019][2917196.504526] Lustre: Skipped 3 previous similar messages [Fri Apr 12 05:31:27 2019][2917208.485463] Lustre: fir-MDT0002: haven't heard from client f5f6ba85-d55c-3558-68dc-e8526225713b (at 10.8.26.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7cfefe8c00, cur 1555072287 expire 1555072137 last 1555072060 [Fri Apr 12 05:31:27 2019][2917208.507346] Lustre: Skipped 3 previous similar messages [Fri Apr 12 05:38:34 2019][2917635.883817] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.20@o2ib6) [Fri Apr 12 05:38:34 2019][2917635.891215] Lustre: Skipped 2244 previous similar messages [Fri Apr 12 05:38:45 2019][2917646.985425] Lustre: fir-MDT0002: Client 777290b7-3360-8b49-8dc3-69b0aa79af92 (at 10.8.17.22@o2ib6) reconnecting [Fri Apr 12 05:38:45 2019][2917646.995692] Lustre: Skipped 2240 previous similar messages [Fri Apr 12 05:45:28 2019][2918049.499998] Lustre: fir-MDT0000: haven't heard from client 00143424-5ee1-ef91-c815-2696e7cd0f21 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5707c6fc00, cur 1555073128 expire 1555072978 last 1555072901 [Fri Apr 12 05:48:34 2019][2918235.903319] Lustre: fir-MDT0002: Connection restored to (at 10.8.18.9@o2ib6) [Fri Apr 12 05:48:34 2019][2918235.910633] Lustre: Skipped 2245 previous similar messages [Fri Apr 12 05:48:41 2019][2918242.887813] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 05:48:45 2019][2918247.084634] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Fri Apr 12 05:48:45 2019][2918247.094734] Lustre: Skipped 2248 previous similar messages [Fri Apr 12 05:58:34 2019][2918836.110490] Lustre: fir-MDT0002: Connection restored to a98aeccc-0553-403d-5376-03640bbd1236 (at 10.9.107.61@o2ib4) [Fri Apr 12 05:58:34 2019][2918836.121116] Lustre: Skipped 2249 previous similar messages [Fri Apr 12 05:58:45 2019][2918847.227547] Lustre: fir-MDT0002: Client b14dc57b-b6ae-736a-c8c8-b28de0f48811 (at 10.9.104.57@o2ib4) reconnecting [Fri Apr 12 05:58:45 2019][2918847.237897] Lustre: Skipped 2236 previous similar messages [Fri Apr 12 06:04:42 2019][2919203.523393] Lustre: fir-MDT0000: haven't heard from client 0724d7cf-c208-4620-633d-b46beeb49d17 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b712eb15400, cur 1555074282 expire 1555074132 last 1555074055 [Fri Apr 12 06:04:42 2019][2919203.545297] Lustre: Skipped 4 previous similar messages [Fri Apr 12 06:07:37 2019][2919378.527156] Lustre: fir-MDT0001: haven't heard from client d1da0c83-1749-86bb-fe69-a46ff18c8c85 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5634a44c00, cur 1555074457 expire 1555074307 last 1555074230 [Fri Apr 12 06:07:37 2019][2919378.548950] Lustre: Skipped 4 previous similar messages [Fri Apr 12 06:07:43 2019][2919384.529598] Lustre: MGS: haven't heard from client c288372f-00ce-d39f-7389-5491b9ccb330 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b76f2b86800, cur 1555074463 expire 1555074313 last 1555074236 [Fri Apr 12 06:07:43 2019][2919384.550706] Lustre: Skipped 2 previous similar messages [Fri Apr 12 06:07:46 2019][2919387.527169] Lustre: fir-MDT0000: haven't heard from client d1da0c83-1749-86bb-fe69-a46ff18c8c85 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b79499e1000, cur 1555074466 expire 1555074316 last 1555074239 [Fri Apr 12 06:08:34 2019][2919436.280278] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.57@o2ib4) [Fri Apr 12 06:08:34 2019][2919436.287781] Lustre: Skipped 2246 previous similar messages [Fri Apr 12 06:08:45 2019][2919447.335836] Lustre: fir-MDT0002: Client ccbe839d-1880-0660-8c5e-d6ce0f68a97f (at 10.9.108.31@o2ib4) reconnecting [Fri Apr 12 06:08:45 2019][2919447.346192] Lustre: Skipped 2241 previous similar messages [Fri Apr 12 06:18:35 2019][2920036.535730] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.68@o2ib4) [Fri Apr 12 06:18:35 2019][2920036.543223] Lustre: Skipped 2243 previous similar messages [Fri Apr 12 06:18:46 2019][2920048.478264] Lustre: fir-MDT0002: Client e18a93c6-a11b-0b52-af65-1483de6beb82 (at 10.8.8.18@o2ib6) reconnecting [Fri Apr 12 06:18:46 2019][2920048.488442] Lustre: Skipped 2248 previous similar messages [Fri Apr 12 06:28:35 2019][2920637.392788] Lustre: fir-MDT0002: Connection restored to 8fa6e6c1-7182-9fd7-64d1-4ba8f1c7c75d (at 10.9.103.10@o2ib4) [Fri Apr 12 06:28:35 2019][2920637.403404] Lustre: Skipped 2245 previous similar messages [Fri Apr 12 06:28:47 2019][2920648.618623] Lustre: fir-MDT0002: Client b7d5a963-1bd8-0aaa-de83-eb1f005c0685 (at 10.9.102.2@o2ib4) reconnecting [Fri Apr 12 06:28:47 2019][2920648.628896] Lustre: Skipped 2237 previous similar messages [Fri Apr 12 06:29:23 2019][2920684.553471] Lustre: fir-MDT0000: haven't heard from client de9c98ab-cdb4-058e-577a-c71d6ba088a9 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6f45f7e400, cur 1555075763 expire 1555075613 last 1555075536 [Fri Apr 12 06:38:36 2019][2921237.690942] Lustre: fir-MDT0002: Connection restored to 532afc32-103a-9744-cab7-ebc849ed6e04 (at 10.8.1.32@o2ib6) [Fri Apr 12 06:38:36 2019][2921237.701379] Lustre: Skipped 2244 previous similar messages [Fri Apr 12 06:38:47 2019][2921248.756607] Lustre: fir-MDT0002: Client 962434f0-d6c3-24ed-71d6-a085c81ce51e (at 10.9.102.24@o2ib4) reconnecting [Fri Apr 12 06:38:47 2019][2921248.766959] Lustre: Skipped 2243 previous similar messages [Fri Apr 12 06:40:20 2019][2921341.566809] Lustre: fir-MDT0000: haven't heard from client 93b0baed-b9ca-bb31-4d5d-79ec1069a835 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5a56bad800, cur 1555076420 expire 1555076270 last 1555076193 [Fri Apr 12 06:40:20 2019][2921341.588699] Lustre: Skipped 4 previous similar messages [Fri Apr 12 06:48:36 2019][2921837.727203] Lustre: fir-MDT0002: Connection restored to (at 10.8.7.13@o2ib6) [Fri Apr 12 06:48:36 2019][2921837.734531] Lustre: Skipped 2249 previous similar messages [Fri Apr 12 06:48:47 2019][2921848.983843] Lustre: fir-MDT0002: Client 2e7834df-4858-b7f9-7a1a-d9dc0f89f5e0 (at 10.9.108.58@o2ib4) reconnecting [Fri Apr 12 06:48:47 2019][2921848.994191] Lustre: Skipped 2245 previous similar messages [Fri Apr 12 06:58:36 2019][2922438.106286] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.58@o2ib4) [Fri Apr 12 06:58:36 2019][2922438.113776] Lustre: Skipped 2244 previous similar messages [Fri Apr 12 06:58:47 2019][2922449.410192] Lustre: fir-MDT0002: Client f419f2f7-2340-142c-2553-5bd564397f99 (at 10.8.28.6@o2ib6) reconnecting [Fri Apr 12 06:58:47 2019][2922449.420410] Lustre: Skipped 2237 previous similar messages [Fri Apr 12 07:08:36 2019][2923038.401027] Lustre: fir-MDT0002: Connection restored to 9af20acb-4644-66cf-ffdb-64cc6425272a (at 10.9.107.28@o2ib4) [Fri Apr 12 07:08:36 2019][2923038.411634] Lustre: Skipped 2237 previous similar messages [Fri Apr 12 07:08:47 2019][2923049.540517] Lustre: fir-MDT0002: Client 58899832-571e-4a18-664b-6f5c1308e27e (at 10.8.18.18@o2ib6) reconnecting [Fri Apr 12 07:08:47 2019][2923049.550782] Lustre: Skipped 2251 previous similar messages [Fri Apr 12 07:18:37 2019][2923638.803337] Lustre: fir-MDT0002: Connection restored to 9ed1e7ae-df1d-ebbe-9d90-d4fc5fa75691 (at 10.9.107.30@o2ib4) [Fri Apr 12 07:18:37 2019][2923638.813949] Lustre: Skipped 2254 previous similar messages [Fri Apr 12 07:18:47 2019][2923649.567332] Lustre: fir-MDT0002: Client 48186cb7-0668-1018-7ab1-1555c19d856e (at 10.9.107.23@o2ib4) reconnecting [Fri Apr 12 07:18:47 2019][2923649.577679] Lustre: Skipped 2236 previous similar messages [Fri Apr 12 07:28:37 2019][2924239.210737] Lustre: fir-MDT0002: Connection restored to 48747558-7da6-0440-a86c-cce04c28222a (at 10.8.7.22@o2ib6) [Fri Apr 12 07:28:37 2019][2924239.221172] Lustre: Skipped 2235 previous similar messages [Fri Apr 12 07:28:48 2019][2924249.891267] Lustre: fir-MDT0002: Client 930cf174-e8e0-277a-21bd-b711955e5992 (at 10.8.1.19@o2ib6) reconnecting [Fri Apr 12 07:28:48 2019][2924249.901448] Lustre: Skipped 2236 previous similar messages [Fri Apr 12 07:33:23 2019][2924525.265244] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 07:38:38 2019][2924840.055282] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.2@o2ib6) [Fri Apr 12 07:38:38 2019][2924840.062520] Lustre: Skipped 2237 previous similar messages [Fri Apr 12 07:38:48 2019][2924850.060279] Lustre: fir-MDT0002: Client 563d53a1-cc0d-a13f-4381-648a7da03caa (at 10.9.105.60@o2ib4) reconnecting [Fri Apr 12 07:38:48 2019][2924850.070635] Lustre: Skipped 2249 previous similar messages [Fri Apr 12 07:48:38 2019][2925440.166051] Lustre: fir-MDT0002: Connection restored to (at 10.9.107.48@o2ib4) [Fri Apr 12 07:48:38 2019][2925440.173544] Lustre: Skipped 2253 previous similar messages [Fri Apr 12 07:48:48 2019][2925450.138089] Lustre: fir-MDT0002: Client 3d0bb7bb-d871-200a-0553-40fe5b229159 (at 10.9.103.11@o2ib4) reconnecting [Fri Apr 12 07:48:48 2019][2925450.148453] Lustre: Skipped 2237 previous similar messages [Fri Apr 12 07:58:38 2019][2926040.191023] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.57@o2ib4) [Fri Apr 12 07:58:38 2019][2926040.198516] Lustre: Skipped 2235 previous similar messages [Fri Apr 12 07:58:48 2019][2926050.205601] Lustre: fir-MDT0002: Client 5242fb43-250e-32ac-e1fd-ee4be217ce12 (at 10.9.106.47@o2ib4) reconnecting [Fri Apr 12 07:58:48 2019][2926050.215962] Lustre: Skipped 2243 previous similar messages [Fri Apr 12 08:08:38 2019][2926640.368821] Lustre: fir-MDT0002: Connection restored to 5cd7ab66-a0ea-0187-060f-51331ba2a50e (at 10.9.108.18@o2ib4) [Fri Apr 12 08:08:38 2019][2926640.379440] Lustre: Skipped 2241 previous similar messages [Fri Apr 12 08:08:49 2019][2926651.149954] Lustre: fir-MDT0002: Client 8b903db2-c3fd-349f-174f-3f69d9c7500b (at 10.9.107.28@o2ib4) reconnecting [Fri Apr 12 08:08:49 2019][2926651.160301] Lustre: Skipped 2246 previous similar messages [Fri Apr 12 08:09:31 2019][2926692.672146] Lustre: fir-MDT0000: haven't heard from client 3e590d56-843b-89cb-7452-aa49631cc59a (at 10.8.25.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b4cc9c6ec00, cur 1555081771 expire 1555081621 last 1555081544 [Fri Apr 12 08:09:31 2019][2926692.694044] Lustre: Skipped 4 previous similar messages [Fri Apr 12 08:09:55 2019][2926716.675513] Lustre: fir-MDT0002: haven't heard from client 3e590d56-843b-89cb-7452-aa49631cc59a (at 10.8.25.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b79fc5de400, cur 1555081795 expire 1555081645 last 1555081568 [Fri Apr 12 08:09:55 2019][2926716.697406] Lustre: Skipped 3 previous similar messages [Fri Apr 12 08:18:39 2019][2927241.100427] Lustre: fir-MDT0002: Connection restored to 0abd045d-746e-a7a6-f1ec-e59a96ee02da (at 10.9.101.64@o2ib4) [Fri Apr 12 08:18:39 2019][2927241.111038] Lustre: Skipped 2246 previous similar messages [Fri Apr 12 08:18:49 2019][2927251.349712] Lustre: fir-MDT0002: Client 45f27c78-9a7f-da99-6869-a1357a032459 (at 10.8.27.8@o2ib6) reconnecting [Fri Apr 12 08:18:49 2019][2927251.359892] Lustre: Skipped 2241 previous similar messages [Fri Apr 12 08:28:39 2019][2927841.175760] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.24@o2ib6) [Fri Apr 12 08:28:39 2019][2927841.183089] Lustre: Skipped 2239 previous similar messages [Fri Apr 12 08:28:49 2019][2927851.430839] Lustre: fir-MDT0002: Client b477f0c2-cfda-70b4-919f-fe195c63921e (at 10.8.7.22@o2ib6) reconnecting [Fri Apr 12 08:28:49 2019][2927851.441023] Lustre: Skipped 2241 previous similar messages [Fri Apr 12 08:34:02 2019][2928163.700853] Lustre: fir-MDT0000: haven't heard from client d3470897-c3f7-cb2f-8e9c-ca0ca58475bc (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5535511000, cur 1555083242 expire 1555083092 last 1555083015 [Fri Apr 12 08:38:39 2019][2928441.454235] Lustre: fir-MDT0002: Connection restored to 15562cac-eb22-f844-7b88-546ba9ddee38 (at 10.9.101.71@o2ib4) [Fri Apr 12 08:38:39 2019][2928441.464848] Lustre: Skipped 2251 previous similar messages [Fri Apr 12 08:38:49 2019][2928451.622666] Lustre: fir-MDT0002: Client 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4) reconnecting [Fri Apr 12 08:38:49 2019][2928451.633030] Lustre: Skipped 2241 previous similar messages [Fri Apr 12 08:40:20 2019][2928541.708069] Lustre: fir-MDT0000: haven't heard from client c2f3ba78-897e-9112-0262-c260aa28a980 (at 10.8.26.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b52b42e9400, cur 1555083620 expire 1555083470 last 1555083393 [Fri Apr 12 08:40:20 2019][2928541.729946] Lustre: Skipped 4 previous similar messages [Fri Apr 12 08:40:30 2019][2928551.709063] Lustre: fir-MDT0002: haven't heard from client c2f3ba78-897e-9112-0262-c260aa28a980 (at 10.8.26.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b73b67e9c00, cur 1555083630 expire 1555083480 last 1555083403 [Fri Apr 12 08:40:30 2019][2928551.730945] Lustre: Skipped 2 previous similar messages [Fri Apr 12 08:48:40 2019][2929041.790247] Lustre: fir-MDT0002: Connection restored to (at 10.9.103.11@o2ib4) [Fri Apr 12 08:48:40 2019][2929041.797734] Lustre: Skipped 2243 previous similar messages [Fri Apr 12 08:48:50 2019][2929051.789829] Lustre: fir-MDT0002: Client 8edb63b6-d814-54d5-ddee-ddb62f3308b6 (at 10.9.103.42@o2ib4) reconnecting [Fri Apr 12 08:48:50 2019][2929051.800187] Lustre: Skipped 2241 previous similar messages [Fri Apr 12 08:57:32 2019][2929573.878230] LustreError: 118731:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b8:0x9752:0x0] [Fri Apr 12 08:57:45 2019][2929587.191741] LustreError: 119003:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b6280e86050 x1628593858357280/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:21/0 lens 488/448 e 0 to 0 dl 1555084671 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 08:57:45 2019][2929587.216240] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 08:57:53 2019][2929594.728268] Lustre: fir-MDT0002: haven't heard from client ddcbdf35-6b0b-042e-5548-90631c5f9cec (at 10.9.106.37@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b72ccf7cc00, cur 1555084673 expire 1555084523 last 1555084446 [Fri Apr 12 08:57:53 2019][2929594.750262] Lustre: Skipped 1 previous similar message [Fri Apr 12 08:58:01 2019][2929602.741699] Lustre: fir-MDT0001: haven't heard from client ddcbdf35-6b0b-042e-5548-90631c5f9cec (at 10.9.106.37@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b59df80b000, cur 1555084681 expire 1555084531 last 1555084454 [Fri Apr 12 08:58:01 2019][2929602.763690] Lustre: Skipped 3 previous similar messages [Fri Apr 12 08:58:40 2019][2929642.083271] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.47@o2ib4) [Fri Apr 12 08:58:40 2019][2929642.090778] Lustre: Skipped 2243 previous similar messages [Fri Apr 12 08:58:50 2019][2929651.870776] Lustre: fir-MDT0002: Client 3405c5b1-22cb-6a4f-06a2-c7a3f21ad911 (at 10.9.107.71@o2ib4) reconnecting [Fri Apr 12 08:58:50 2019][2929651.881158] Lustre: Skipped 2247 previous similar messages [Fri Apr 12 08:59:49 2019][2929711.138318] LustreError: 118955:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b4d0085cc50 x1628593858714928/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:25/0 lens 488/448 e 0 to 0 dl 1555084795 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 08:59:49 2019][2929711.163801] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 09:00:11 2019][2929732.708937] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:06:06 2019][2930087.713172] LustreError: 118666:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b8:0x9ae3:0x0] [Fri Apr 12 09:06:54 2019][2930136.112877] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:08:40 2019][2930242.137925] Lustre: fir-MDT0002: Connection restored to d6d72e8f-7940-a8bd-6b0b-3c9d0477872e (at 10.8.2.33@o2ib6) [Fri Apr 12 09:08:40 2019][2930242.148365] Lustre: Skipped 2247 previous similar messages [Fri Apr 12 09:08:50 2019][2930252.031999] Lustre: fir-MDT0002: Client e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6) reconnecting [Fri Apr 12 09:08:50 2019][2930252.042197] Lustre: Skipped 2238 previous similar messages [Fri Apr 12 09:15:15 2019][2930637.491830] Lustre: 118324:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 09:15:15 2019][2930637.491830] req@ffff8b63366d6c00 x1630377110342896/t146512056348(0) o36->777290b7-3360-8b49-8dc3-69b0aa79af92@10.8.17.22@o2ib6:20/0 lens 512/2888 e 1 to 0 dl 1555085720 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:15:15 2019][2930637.521909] Lustre: 118324:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Fri Apr 12 09:16:30 2019][2930712.460296] LustreError: 118171:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555085700, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b7023bb0900/0x53d0f22265ce875c lrc: 3/1,0 mode: --/PR res: [0x2c0012964:0x42f6:0x0].0x0 bits 0x13/0x8 rrc: 9 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118171 timeout: 0 lvb_type: 0 [Fri Apr 12 09:16:30 2019][2930712.500047] LustreError: 118171:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Fri Apr 12 09:18:21 2019][2930822.993478] LNet: Service thread pid 118276 was inactive for 200.53s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 09:18:21 2019][2930823.010886] Pid: 118276, comm: mdt01_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 09:18:21 2019][2930823.020921] Call Trace: [Fri Apr 12 09:18:21 2019][2930823.023566] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 09:18:21 2019][2930823.030700] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 09:18:21 2019][2930823.038078] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 09:18:21 2019][2930823.045038] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 09:18:21 2019][2930823.051694] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 09:18:21 2019][2930823.057933] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 09:18:21 2019][2930823.064675] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 09:18:21 2019][2930823.070568] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 09:18:21 2019][2930823.077695] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 09:18:21 2019][2930823.085599] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 09:18:21 2019][2930823.092116] [] kthread+0xd1/0xe0 [Fri Apr 12 09:18:21 2019][2930823.097203] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 09:18:21 2019][2930823.103870] [] 0xffffffffffffffff [Fri Apr 12 09:18:21 2019][2930823.109087] LustreError: dumping log to /tmp/lustre-log.1555085901.118276 [Fri Apr 12 09:18:22 2019][2930824.698896] LNet: Service thread pid 118171 was inactive for 202.23s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 09:18:22 2019][2930824.717413] Pid: 118171, comm: mdt02_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 09:18:22 2019][2930824.727431] Call Trace: [Fri Apr 12 09:18:22 2019][2930824.730070] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 09:18:22 2019][2930824.737192] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 09:18:22 2019][2930824.744565] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 09:18:23 2019][2930824.751568] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 09:18:23 2019][2930824.758746] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 09:18:23 2019][2930824.765409] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 09:18:23 2019][2930824.772585] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 09:18:23 2019][2930824.779326] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 09:18:23 2019][2930824.785986] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 09:18:23 2019][2930824.792936] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 09:18:23 2019][2930824.800217] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 09:18:23 2019][2930824.806567] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 09:18:23 2019][2930824.813694] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 09:18:23 2019][2930824.821580] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 09:18:23 2019][2930824.828100] [] kthread+0xd1/0xe0 [Fri Apr 12 09:18:23 2019][2930824.833201] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 09:18:23 2019][2930824.839849] [] 0xffffffffffffffff [Fri Apr 12 09:18:40 2019][2930842.227906] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.26@o2ib4) [Fri Apr 12 09:18:40 2019][2930842.235420] Lustre: Skipped 2258 previous similar messages [Fri Apr 12 09:18:50 2019][2930852.370057] Lustre: fir-MDT0002: Client 74806a48-c1e8-fa00-f93a-73121ee183ec (at 10.9.103.43@o2ib4) reconnecting [Fri Apr 12 09:18:50 2019][2930852.380408] Lustre: Skipped 2251 previous similar messages [Fri Apr 12 09:19:48 2019][2930910.383070] Lustre: 119200:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629749682394512 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b5f096d1450 x1629749682394512/t0(0) o4->a132911f-e28d-2e04-1263-47066608ec3c@10.8.23.24@o2ib6:18/0 lens 2384/0 e 0 to 0 dl 1555086018 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 09:19:48 2019][2930910.412832] Lustre: 119200:0:(tgt_handler.c:562:tgt_handle_recovery()) Skipped 2 previous similar messages [Fri Apr 12 09:21:04 2019][2930986.194695] Lustre: 117969:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 09:21:04 2019][2930986.194695] req@ffff8b7740f0e000 x1628752834278640/t189030044486(0) o36->338dce11-1d8c-d11c-4363-af216d5b0418@10.9.108.48@o2ib4:9/0 lens 512/2888 e 0 to 0 dl 1555086069 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:21:04 2019][2930986.225213] Lustre: 117969:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages [Fri Apr 12 09:22:09 2019][2931051.142983] LustreError: 118560:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555086039, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0003_UUID lock: ffff8b855f6b2d00/0x53d0f22272ad3db6 lrc: 3/0,1 mode: --/PW res: [0x2800128e2:0x73b:0x0].0x0 bits 0x40/0x0 rrc: 6 type: IBT flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 118560 timeout: 0 lvb_type: 0 [Fri Apr 12 09:22:09 2019][2931051.182653] LustreError: 118560:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Fri Apr 12 09:22:33 2019][2931075.443265] LustreError: 119004:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b5925052450 x1628593861763968/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:3/0 lens 488/448 e 0 to 0 dl 1555086183 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:22:33 2019][2931075.467671] LustreError: 119004:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 8 previous similar messages [Fri Apr 12 09:22:33 2019][2931075.477348] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 09:22:33 2019][2931075.490668] Lustre: Skipped 8 previous similar messages [Fri Apr 12 09:23:59 2019][2931161.432181] LNet: Service thread pid 118560 was inactive for 200.28s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 09:23:59 2019][2931161.449382] Pid: 118560, comm: mdt00_105 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 09:23:59 2019][2931161.459383] Call Trace: [Fri Apr 12 09:23:59 2019][2931161.462029] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 09:23:59 2019][2931161.469156] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 09:23:59 2019][2931161.476533] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 09:23:59 2019][2931161.483465] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 09:23:59 2019][2931161.490121] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 09:23:59 2019][2931161.496340] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 09:23:59 2019][2931161.503083] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 09:23:59 2019][2931161.508962] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 09:23:59 2019][2931161.516094] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 09:23:59 2019][2931161.523990] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 09:23:59 2019][2931161.530499] [] kthread+0xd1/0xe0 [Fri Apr 12 09:23:59 2019][2931161.535586] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 09:23:59 2019][2931161.542234] [] 0xffffffffffffffff [Fri Apr 12 09:23:59 2019][2931161.547427] LustreError: dumping log to /tmp/lustre-log.1555086239.118560 [Fri Apr 12 09:24:00 2019][2931162.701811] Pid: 118246, comm: mdt03_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 09:24:00 2019][2931162.711815] Call Trace: [Fri Apr 12 09:24:00 2019][2931162.714461] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 09:24:00 2019][2931162.721585] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 09:24:00 2019][2931162.728960] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 09:24:00 2019][2931162.735963] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 09:24:00 2019][2931162.743139] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 09:24:01 2019][2931162.749796] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 09:24:01 2019][2931162.756971] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 09:24:01 2019][2931162.763715] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 09:24:01 2019][2931162.770373] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 09:24:01 2019][2931162.777305] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 09:24:01 2019][2931162.784594] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 09:24:01 2019][2931162.790939] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 09:24:01 2019][2931162.798063] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 09:24:01 2019][2931162.805958] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 09:24:01 2019][2931162.812466] [] kthread+0xd1/0xe0 [Fri Apr 12 09:24:01 2019][2931162.817553] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 09:24:01 2019][2931162.824202] [] 0xffffffffffffffff [Fri Apr 12 09:24:10 2019][2931172.154965] LustreError: 118957:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b7ca8f44c50 x1629297487313408/t0(0) o4->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:16/0 lens 488/448 e 0 to 0 dl 1555086256 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:24:10 2019][2931172.179387] LustreError: 118957:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 3 previous similar messages [Fri Apr 12 09:24:10 2019][2931172.189071] Lustre: fir-MDT0002: Bulk IO write error with a77148c9-d434-623d-b422-9d417c28aa4b (at 10.8.1.32@o2ib6), client will retry: rc = -107 [Fri Apr 12 09:24:10 2019][2931172.202292] Lustre: Skipped 2 previous similar messages [Fri Apr 12 09:25:36 2019][2931257.807837] LustreError: 118951:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b6e0eed6450 x1629289980231184/t0(0) o4->bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83@10.8.26.21@o2ib6:6/0 lens 488/448 e 0 to 0 dl 1555086366 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:25:36 2019][2931257.814602] Lustre: fir-MDT0002: Bulk IO write error with bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83 (at 10.8.26.21@o2ib6), client will retry: rc = -107 [Fri Apr 12 09:25:36 2019][2931257.826097] Lustre: 118903:0:(tgt_handler.c:562:tgt_handle_recovery()) @@@ rq_xid 1629289980231360 matches saved xid, expected REPLAY or RESENT flag (0) req@ffff8b71bcb0ec50 x1629289980231360/t0(0) o4->bfeb95bd-cdaa-d03f-0d56-9a8ff3bc2f83@10.8.26.21@o2ib6:6/0 lens 1016/0 e 0 to 0 dl 1555086366 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [Fri Apr 12 09:25:36 2019][2931257.875148] LustreError: 118951:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 2 previous similar messages [Fri Apr 12 09:25:56 2019][2931278.074898] LNetError: 117128:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:26:58 2019][2931339.762828] Lustre: fir-MDT0001: haven't heard from client 7a475994-4fa5-4f3b-5ff1-5f6e376a0449 (at 10.8.25.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5b117c8000, cur 1555086418 expire 1555086268 last 1555086191 [Fri Apr 12 09:27:07 2019][2931348.762957] Lustre: fir-MDT0002: haven't heard from client 7a475994-4fa5-4f3b-5ff1-5f6e376a0449 (at 10.8.25.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7cfefe9400, cur 1555086427 expire 1555086277 last 1555086200 [Fri Apr 12 09:27:07 2019][2931348.785055] Lustre: Skipped 3 previous similar messages [Fri Apr 12 09:27:31 2019][2931373.331011] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:27:38 2019][2931380.351118] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:28:40 2019][2931442.515564] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.22@o2ib6) [Fri Apr 12 09:28:40 2019][2931442.522885] Lustre: Skipped 2383 previous similar messages [Fri Apr 12 09:28:52 2019][2931453.965467] Lustre: fir-MDT0002: Client 563d53a1-cc0d-a13f-4381-648a7da03caa (at 10.9.105.60@o2ib4) reconnecting [Fri Apr 12 09:28:52 2019][2931453.975812] Lustre: Skipped 2307 previous similar messages [Fri Apr 12 09:29:44 2019][2931506.674918] LNetError: 117116:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:33:03 2019][2931705.450691] LNetError: 117122:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:33:22 2019][2931724.229477] LNetError: 117129:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:33:39 2019][2931741.208108] LustreError: 118642:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010808:0xd6ae:0x0] [Fri Apr 12 09:34:06 2019][2931768.124023] LustreError: 119005:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00128f9:0x404d:0x0] [Fri Apr 12 09:34:14 2019][2931775.771345] Lustre: fir-MDT0002: haven't heard from client 48668d7e-41a9-a375-0254-f129c181cad6 (at 10.8.20.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7cfefeb000, cur 1555086854 expire 1555086704 last 1555086627 [Fri Apr 12 09:36:08 2019][2931890.630934] LustreError: 119078:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b8:0xa2b8:0x0] [Fri Apr 12 09:37:01 2019][2931943.534007] LustreError: 118929:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b697cb31450 x1628593864939232/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:7/0 lens 488/448 e 0 to 0 dl 1555087027 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:37:01 2019][2931943.558482] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 09:37:01 2019][2931943.571811] Lustre: Skipped 2 previous similar messages [Fri Apr 12 09:37:36 2019][2931977.837886] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:38:25 2019][2932026.812229] LustreError: 119200:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b60db93c850 x1630377532578080/t0(0) o4->e4292209-c55b-cc00-a078-2d504f5a17aa@10.8.1.24@o2ib6:25/0 lens 488/448 e 0 to 0 dl 1555087135 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:38:25 2019][2932026.815797] Lustre: fir-MDT0002: Bulk IO write error with e4292209-c55b-cc00-a078-2d504f5a17aa (at 10.8.1.24@o2ib6), client will retry: rc = -107 [Fri Apr 12 09:38:25 2019][2932026.850129] LustreError: 119200:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 1 previous similar message [Fri Apr 12 09:38:41 2019][2932042.931382] Lustre: fir-MDT0002: Connection restored to (at 10.8.11.26@o2ib6) [Fri Apr 12 09:38:41 2019][2932042.938836] Lustre: Skipped 2306 previous similar messages [Fri Apr 12 09:38:52 2019][2932054.123563] Lustre: fir-MDT0002: Client 99abaf9a-6d16-bfe7-2d5c-9e199780b8b7 (at 10.9.101.71@o2ib4) reconnecting [Fri Apr 12 09:38:52 2019][2932054.133936] Lustre: Skipped 2295 previous similar messages [Fri Apr 12 09:39:44 2019][2932106.317237] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:41:22 2019][2932204.613295] LustreError: 119117:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0003: BRW to missing obj [0x280012881:0x5d68:0x0] [Fri Apr 12 09:44:37 2019][2932399.744950] LustreError: 118609:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b8:0xa5b5:0x0] [Fri Apr 12 09:45:48 2019][2932470.690153] LustreError: 118733:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b7ca8f44c50 x1628593866282704/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:18/0 lens 488/448 e 0 to 0 dl 1555087578 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:45:48 2019][2932470.714667] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 09:45:48 2019][2932470.727973] Lustre: Skipped 1 previous similar message [Fri Apr 12 09:46:45 2019][2932527.485216] LNetError: 117130:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:48:41 2019][2932642.998485] Lustre: fir-MDT0002: Connection restored to (at 10.8.27.6@o2ib6) [Fri Apr 12 09:48:41 2019][2932643.005806] Lustre: Skipped 2295 previous similar messages [Fri Apr 12 09:48:52 2019][2932654.293225] Lustre: fir-MDT0002: Client 777290b7-3360-8b49-8dc3-69b0aa79af92 (at 10.8.17.22@o2ib6) reconnecting [Fri Apr 12 09:48:52 2019][2932654.303543] Lustre: Skipped 2297 previous similar messages [Fri Apr 12 09:49:54 2019][2932716.259316] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:49:54 2019][2932716.271849] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages [Fri Apr 12 09:55:02 2019][2933023.866072] LustreError: 118652:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00111aa:0x48f9:0x0] [Fri Apr 12 09:55:38 2019][2933059.796235] Lustre: fir-MDT0000: haven't heard from client e86a4b7f-ef5b-1013-9146-8b0edcdaec62 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b7960242800, cur 1555088138 expire 1555087988 last 1555087911 [Fri Apr 12 09:55:38 2019][2933059.818138] Lustre: Skipped 4 previous similar messages [Fri Apr 12 09:55:52 2019][2933074.130618] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 09:55:52 2019][2933074.143157] LNetError: 117121:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages [Fri Apr 12 09:56:33 2019][2933115.474865] LustreError: 118811:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001097d:0x7287:0x0] [Fri Apr 12 09:56:40 2019][2933121.911270] LustreError: 118835:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b6018cefc50 x1628593868051168/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:9/0 lens 488/448 e 0 to 0 dl 1555088229 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 09:56:40 2019][2933121.935760] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 09:58:41 2019][2933243.051377] Lustre: fir-MDT0002: Connection restored to (at 10.9.108.11@o2ib4) [Fri Apr 12 09:58:41 2019][2933243.059002] Lustre: Skipped 2309 previous similar messages [Fri Apr 12 09:58:52 2019][2933254.734687] Lustre: fir-MDT0002: Client 5242fb43-250e-32ac-e1fd-ee4be217ce12 (at 10.9.106.47@o2ib4) reconnecting [Fri Apr 12 09:58:52 2019][2933254.745075] Lustre: Skipped 2308 previous similar messages [Fri Apr 12 10:01:40 2019][2933422.036218] Lustre: 117551:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 10:01:40 2019][2933422.036218] req@ffff8b58aed1ce00 x1629299675627776/t0(0) o101->d1633d28-3ef3-75ae-ab7a-2bfb540fe368@10.8.18.14@o2ib6:15/0 lens 576/3264 e 0 to 0 dl 1555088505 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 10:02:45 2019][2933486.835476] LustreError: 118552:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555088475, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b4f36f29b00/0x53d0f222f52456a2 lrc: 3/1,0 mode: --/PR res: [0x2c001065f:0x36f2:0x0].0x0 bits 0x13/0x8 rrc: 12 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118552 timeout: 0 lvb_type: 0 [Fri Apr 12 10:02:45 2019][2933486.875571] LustreError: 118552:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Fri Apr 12 10:04:35 2019][2933597.063566] LNet: Service thread pid 118552 was inactive for 200.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 10:04:35 2019][2933597.080769] LNet: Skipped 1 previous similar message [Fri Apr 12 10:04:35 2019][2933597.085933] Pid: 118552, comm: mdt00_098 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 10:04:35 2019][2933597.095955] Call Trace: [Fri Apr 12 10:04:35 2019][2933597.098608] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 10:04:35 2019][2933597.105738] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 10:04:35 2019][2933597.113165] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 10:04:35 2019][2933597.120183] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 10:04:35 2019][2933597.127430] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 10:04:35 2019][2933597.134111] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 10:04:35 2019][2933597.141305] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 10:04:35 2019][2933597.148064] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 10:04:35 2019][2933597.154746] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 10:04:35 2019][2933597.161705] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 10:04:35 2019][2933597.169012] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 10:04:35 2019][2933597.175391] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 10:04:35 2019][2933597.182554] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 10:04:35 2019][2933597.190484] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 10:04:35 2019][2933597.197025] [] kthread+0xd1/0xe0 [Fri Apr 12 10:04:35 2019][2933597.202127] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 10:04:35 2019][2933597.208776] [] 0xffffffffffffffff [Fri Apr 12 10:04:35 2019][2933597.213989] LustreError: dumping log to /tmp/lustre-log.1555088675.118552 [Fri Apr 12 10:04:36 2019][2933598.691093] LNet: Service thread pid 118551 was inactive for 201.85s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 10:04:36 2019][2933598.708317] Pid: 118551, comm: mdt03_056 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 10:04:36 2019][2933598.718321] Call Trace: [Fri Apr 12 10:04:36 2019][2933598.720959] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 10:04:36 2019][2933598.728082] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 10:04:36 2019][2933598.735459] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 10:04:36 2019][2933598.742383] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 10:04:36 2019][2933598.749038] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 10:04:36 2019][2933598.755260] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 10:04:36 2019][2933598.762008] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 10:04:36 2019][2933598.767887] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 10:04:36 2019][2933598.775031] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 10:04:36 2019][2933598.782926] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 10:04:36 2019][2933598.789441] [] kthread+0xd1/0xe0 [Fri Apr 12 10:04:36 2019][2933598.794531] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 10:04:36 2019][2933598.801178] [] 0xffffffffffffffff [Fri Apr 12 10:05:19 2019][2933641.166313] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 10:05:19 2019][2933641.179279] LNetError: 117127:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 15 previous similar messages [Fri Apr 12 10:08:02 2019][2933803.815270] LustreError: 119078:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b60db93f050 x1628593869542128/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:1/0 lens 488/448 e 0 to 0 dl 1555088911 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 10:08:02 2019][2933803.839809] LustreError: 119078:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 1 previous similar message [Fri Apr 12 10:08:02 2019][2933803.849492] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 10:08:02 2019][2933803.862863] Lustre: Skipped 1 previous similar message [Fri Apr 12 10:08:41 2019][2933843.145707] Lustre: fir-MDT0002: Connection restored to a98aeccc-0553-403d-5376-03640bbd1236 (at 10.9.107.61@o2ib4) [Fri Apr 12 10:08:41 2019][2933843.156327] Lustre: Skipped 2337 previous similar messages [Fri Apr 12 10:08:53 2019][2933855.148556] Lustre: fir-MDT0002: Client e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6) reconnecting [Fri Apr 12 10:08:53 2019][2933855.158792] Lustre: Skipped 2320 previous similar messages [Fri Apr 12 10:10:10 2019][2933932.108594] LustreError: 117513:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001306b:0x3bb3:0x0] [Fri Apr 12 10:12:13 2019][2934055.272105] LustreError: 118841:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0003: BRW to missing obj [0x280012881:0x840f:0x0] [Fri Apr 12 10:15:24 2019][2934246.262810] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 10:15:24 2019][2934246.276117] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 10 previous similar messages [Fri Apr 12 10:18:41 2019][2934443.464737] Lustre: fir-MDT0002: Connection restored to (at 10.9.104.57@o2ib4) [Fri Apr 12 10:18:41 2019][2934443.472238] Lustre: Skipped 2328 previous similar messages [Fri Apr 12 10:18:53 2019][2934455.305366] Lustre: fir-MDT0002: Client f87bfb7b-5313-2d83-7524-cc6a927681f0 (at 10.9.108.65@o2ib4) reconnecting [Fri Apr 12 10:18:53 2019][2934455.315731] Lustre: Skipped 2345 previous similar messages [Fri Apr 12 10:25:05 2019][2934827.353200] LustreError: 119193:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b621b0ead00 x1630280176512240/t0(0) o37->6ea7ad88-4c41-3a56-d580-e39b0e85132b@10.8.27.6@o2ib6:11/0 lens 448/440 e 0 to 0 dl 1555089911 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 10:25:37 2019][2934858.956411] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 10:25:37 2019][2934858.968992] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 16 previous similar messages [Fri Apr 12 10:28:41 2019][2935043.575355] Lustre: fir-MDT0002: Connection restored to 73a6789a-9f89-2338-37f2-4564b416dba3 (at 10.8.10.32@o2ib6) [Fri Apr 12 10:28:41 2019][2935043.585936] Lustre: Skipped 2346 previous similar messages [Fri Apr 12 10:28:53 2019][2935055.644563] Lustre: fir-MDT0002: Client 3c1359c4-6f8a-28e8-3fe1-a0a63b3ad896 (at 10.9.102.1@o2ib4) reconnecting [Fri Apr 12 10:28:53 2019][2935055.654935] Lustre: Skipped 2336 previous similar messages [Fri Apr 12 10:28:58 2019][2935060.551327] LustreError: 119072:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b60c125bc50 x1629299426801712/t0(0) o4->39b26b25-3151-384d-34d9-da20c3dc7f78@10.8.7.35@o2ib6:28/0 lens 488/448 e 0 to 0 dl 1555090168 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 10:28:58 2019][2935060.576207] Lustre: fir-MDT0002: Bulk IO write error with 39b26b25-3151-384d-34d9-da20c3dc7f78 (at 10.8.7.35@o2ib6), client will retry: rc = -107 [Fri Apr 12 10:28:58 2019][2935060.589760] Lustre: Skipped 1 previous similar message [Fri Apr 12 10:32:30 2019][2935272.542372] LustreError: 119009:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001079b:0x159c:0x0] [Fri Apr 12 10:32:42 2019][2935283.838316] Lustre: fir-MDT0002: haven't heard from client e65e5035-9b3f-f195-f2ef-d9af03a416f4 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b641322b800, cur 1555090362 expire 1555090212 last 1555090135 [Fri Apr 12 10:32:42 2019][2935283.860632] Lustre: Skipped 4 previous similar messages [Fri Apr 12 10:37:08 2019][2935550.347186] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 10:37:08 2019][2935550.360024] LNetError: 117115:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 12 previous similar messages [Fri Apr 12 10:37:26 2019][2935568.346006] LustreError: 118840:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00105d2:0x3109:0x0] [Fri Apr 12 10:37:30 2019][2935571.846919] Lustre: fir-MDT0003: haven't heard from client f419f2f7-2340-142c-2553-5bd564397f99 (at 10.8.28.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b6696ee2400, cur 1555090650 expire 1555090500 last 1555090423 [Fri Apr 12 10:37:30 2019][2935571.869275] Lustre: Skipped 4 previous similar messages [Fri Apr 12 10:38:42 2019][2935644.075745] Lustre: fir-MDT0002: Connection restored to (at 10.8.1.19@o2ib6) [Fri Apr 12 10:38:42 2019][2935644.083119] Lustre: Skipped 2196 previous similar messages [Fri Apr 12 10:38:54 2019][2935656.047191] Lustre: fir-MDT0002: Client 95f309fb-7ae1-b5a5-cb40-881c2341a31d (at 10.9.106.64@o2ib4) reconnecting [Fri Apr 12 10:38:54 2019][2935656.057618] Lustre: Skipped 2094 previous similar messages [Fri Apr 12 10:39:33 2019][2935695.166184] LustreError: 118688:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b60c1258050 x1628593873729408/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:9/0 lens 488/448 e 0 to 0 dl 1555090779 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 10:39:33 2019][2935695.191626] LustreError: 118688:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 2 previous similar messages [Fri Apr 12 10:39:33 2019][2935695.201370] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 10:39:33 2019][2935695.214717] Lustre: Skipped 1 previous similar message [Fri Apr 12 10:43:04 2019][2935906.173854] LustreError: 118982:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001310d:0x1c92:0x0] [Fri Apr 12 10:43:04 2019][2935906.678996] LustreError: 118940:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001079b:0x168a:0x0] [Fri Apr 12 10:43:05 2019][2935906.858685] LustreError: 118940:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00104ca:0x18fe:0x0] [Fri Apr 12 10:43:30 2019][2935932.477477] LustreError: 118982:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b698afd6050 x1630377341815680/t0(0) o4->e47c55d1-16e3-2d00-bb0a-bee0e04bc96f@10.8.1.17@o2ib6:0/0 lens 488/448 e 0 to 0 dl 1555091040 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 10:43:30 2019][2935932.501968] Lustre: fir-MDT0002: Bulk IO write error with e47c55d1-16e3-2d00-bb0a-bee0e04bc96f (at 10.8.1.17@o2ib6), client will retry: rc = -107 [Fri Apr 12 10:47:29 2019][2936171.077881] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 10:47:29 2019][2936171.090439] LNetError: 117124:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 9 previous similar messages [Fri Apr 12 10:48:18 2019][2936219.920910] LustreError: 118687:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0003: BRW to missing obj [0x280012881:0xb148:0x0] [Fri Apr 12 10:48:42 2019][2936244.210390] Lustre: fir-MDT0002: Connection restored to (at 10.9.106.64@o2ib4) [Fri Apr 12 10:48:42 2019][2936244.217933] Lustre: Skipped 1845 previous similar messages [Fri Apr 12 10:48:53 2019][2936254.838867] LustreError: 119186:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b4e162b3f00 x1629297528927888/t0(0) o37->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:22/0 lens 448/440 e 0 to 0 dl 1555091362 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 10:48:54 2019][2936256.432813] Lustre: fir-MDT0002: Client 3e76556a-2309-201a-ab84-b4f3a844a5cf (at 10.9.103.8@o2ib4) reconnecting [Fri Apr 12 10:48:54 2019][2936256.443092] Lustre: Skipped 1846 previous similar messages [Fri Apr 12 10:55:04 2019][2936625.847505] LustreError: 119072:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0012921:0x2ec3:0x0] [Fri Apr 12 10:55:08 2019][2936630.118802] LustreError: 119021:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107f9:0x5de0:0x0] [Fri Apr 12 10:58:08 2019][2936810.427257] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 10:58:08 2019][2936810.439812] LNetError: 117126:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 8 previous similar messages [Fri Apr 12 10:58:42 2019][2936844.241103] Lustre: fir-MDT0002: Connection restored to (at 10.9.107.71@o2ib4) [Fri Apr 12 10:58:42 2019][2936844.248835] Lustre: Skipped 1845 previous similar messages [Fri Apr 12 10:58:45 2019][2936847.376030] LustreError: 118940:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c0010847:0x2201:0x0] [Fri Apr 12 10:58:54 2019][2936856.478952] Lustre: fir-MDT0002: Client 56e578a3-b6d5-f5cd-9f6d-fd84ca1112e5 (at 10.8.8.8@o2ib6) reconnecting [Fri Apr 12 10:58:54 2019][2936856.489080] Lustre: Skipped 1839 previous similar messages [Fri Apr 12 11:02:51 2019][2937093.355817] Lustre: 118196:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 11:02:51 2019][2937093.355817] req@ffff8b5c44f33300 x1629295000843712/t0(0) o101->ef2a72e8-1102-fad1-812c-2f0167c49e1f@10.9.101.67@o2ib4:26/0 lens 576/3264 e 1 to 0 dl 1555092176 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:02:51 2019][2937093.385157] Lustre: 118196:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Fri Apr 12 11:02:52 2019][2937093.857822] Lustre: 118517:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply [Fri Apr 12 11:02:52 2019][2937093.857822] req@ffff8b792171f800 x1629298722351312/t146538816154(0) o36->dbf4d020-f48d-b6b2-3e31-f500fe6cd668@10.9.103.19@o2ib4:26/0 lens 576/2888 e 1 to 0 dl 1555092176 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:03:01 2019][2937102.877993] Lustre: 118374:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:03:01 2019][2937102.877993] req@ffff8b6e71f09e00 x1630379791416976/t0(0) o101->565bcec1-6ed3-f69e-4515-c8cd5474ca24@10.9.104.37@o2ib4:6/0 lens 576/3264 e 0 to 0 dl 1555092186 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:03:05 2019][2937107.286560] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:03:05 2019][2937107.286560] req@ffff8b6556ff0300 x1628659777324912/t0(0) o36->8f523a71-dbaa-8a10-c21b-13f39a091fa7@10.9.102.10@o2ib4:10/0 lens 608/2888 e 0 to 0 dl 1555092190 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:03:05 2019][2937107.316913] Lustre: 118310:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 16 previous similar messages [Fri Apr 12 11:03:12 2019][2937114.022206] Lustre: 118764:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:03:12 2019][2937114.022206] req@ffff8b89fda0d100 x1630456158562256/t0(0) o36->1e6f74c6-3496-d691-5a66-fdc2823a3b7a@10.9.101.37@o2ib4:17/0 lens 856/2888 e 0 to 0 dl 1555092197 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:03:12 2019][2937114.051474] Lustre: 118764:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message [Fri Apr 12 11:03:22 2019][2937124.528414] Lustre: 118173:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:03:22 2019][2937124.528414] req@ffff8b71a02fe600 x1629292287053008/t0(0) o36->95ffd939-3208-0e7f-0f16-2c00789b1623@10.8.27.22@o2ib6:27/0 lens 600/2888 e 0 to 0 dl 1555092207 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:03:22 2019][2937124.557780] Lustre: 118173:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Fri Apr 12 11:03:39 2019][2937140.970747] Lustre: 118349:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:03:39 2019][2937140.970747] req@ffff8b6641c67500 x1628455948133584/t0(0) o36->49bb0e27-60f6-691c-951f-2d5ad66437a7@10.8.11.27@o2ib6:14/0 lens 536/2888 e 0 to 0 dl 1555092224 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:03:39 2019][2937141.000213] Lustre: 118349:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages [Fri Apr 12 11:04:06 2019][2937168.042286] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Fri Apr 12 11:04:06 2019][2937168.057459] LustreError: 118568:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092156, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b783ffbdc40/0x53d0f223f4c7f3b2 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f223f4c7f3b9 expref: -99 pid: 118568 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:06 2019][2937168.097871] LustreError: 118568:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 9 previous similar messages [Fri Apr 12 11:04:06 2019][2937168.103293] LustreError: 118281:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092156, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b5e7e7bc5c0/0x53d0f223f4c8a3fb lrc: 3/1,0 mode: --/PR res: [0x2c0013075:0x743:0x0].0x0 bits 0x13/0x8 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118281 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:06 2019][2937168.148914] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Fri Apr 12 11:04:07 2019][2937169.636323] LustreError: 118300:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092157, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b8841ad7080/0x53d0f223f4d951fb lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 25 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118300 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:07 2019][2937169.675847] LustreError: 118300:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 11 previous similar messages [Fri Apr 12 11:04:10 2019][2937172.500384] LustreError: 117898:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092160, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b61ad5fe300/0x53d0f223f4f900b6 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 26 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 117898 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:11 2019][2937173.300394] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Fri Apr 12 11:04:12 2019][2937174.571446] Lustre: 118526:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:04:12 2019][2937174.571446] req@ffff8b72fc2b5a00 x1628575381213760/t0(0) o36->3faa2379-89f1-5fe8-6bd5-0b0f4e027869@10.9.106.4@o2ib4:17/0 lens 600/2888 e 0 to 0 dl 1555092257 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:04:12 2019][2937174.600615] Lustre: 118526:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages [Fri Apr 12 11:04:21 2019][2937183.395604] LustreError: 118280:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092171, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0002_UUID lock: ffff8b5a761c5c40/0x53d0f223f5762eb6 lrc: 3/1,0 mode: --/PR res: [0x2c00130a5:0x7008:0x0].0x0 bits 0x13/0x0 rrc: 50 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 118280 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:21 2019][2937183.435697] LustreError: 118280:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Fri Apr 12 11:04:27 2019][2937189.547734] LustreError: 118382:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092177, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b609f6821c0/0x53d0f223f5bc880b lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 29 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118382 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:39 2019][2937201.317969] LustreError: 118468:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092189, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b678bac1d40/0x53d0f223f642ec94 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 29 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118468 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:39 2019][2937201.357571] LustreError: 118468:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 1 previous similar message [Fri Apr 12 11:04:44 2019][2937206.694054] LustreError: 117466:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092194, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0002 lock: ffff8b59738a5c40/0x53d0f223f680d18b lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 14 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f223f680e167 expref: -99 pid: 117466 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:44 2019][2937206.734894] LustreError: 117466:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Fri Apr 12 11:04:58 2019][2937220.208367] LustreError: 118199:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092208, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b61b3699680/0x53d0f223f714ee38 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 34 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118199 timeout: 0 lvb_type: 0 [Fri Apr 12 11:04:58 2019][2937220.248537] LustreError: 118199:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Fri Apr 12 11:05:17 2019][2937239.474239] LustreError: 118652:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c00107b8:0xb6f8:0x0] [Fri Apr 12 11:05:20 2019][2937242.444804] Lustre: 118636:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:05:20 2019][2937242.444804] req@ffff8b82002b3300 x1628639838342032/t0(0) o36->7b132c53-d4cd-2af2-1021-d3b19184bca3@10.9.107.44@o2ib4:25/0 lens 560/2888 e 0 to 0 dl 1555092325 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:05:20 2019][2937242.474089] Lustre: 118636:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [Fri Apr 12 11:05:30 2019][2937252.539015] LustreError: 118357:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092240, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b8754366c00/0x53d0f223f875f689 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 54 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118357 timeout: 0 lvb_type: 0 [Fri Apr 12 11:05:30 2019][2937252.578504] LustreError: 118357:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 2 previous similar messages [Fri Apr 12 11:05:37 2019][2937258.916078] LustreError: 119204:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0003: BRW to missing obj [0x280012881:0xc97e:0x0] [Fri Apr 12 11:05:39 2019][2937261.273185] Lustre: fir-MDT0000-osp-MDT0002: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Fri Apr 12 11:05:39 2019][2937261.288488] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Fri Apr 12 11:05:39 2019][2937261.298497] Lustre: Skipped 1 previous similar message [Fri Apr 12 11:05:55 2019][2937277.371515] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Fri Apr 12 11:05:55 2019][2937277.386679] LustreError: 118245:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092265, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0001 lock: ffff8b75f8648240/0x53d0f223f98d7135 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 24 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f223f98d7231 expref: -99 pid: 118245 timeout: 0 lvb_type: 0 [Fri Apr 12 11:05:55 2019][2937277.427024] LustreError: 118245:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Fri Apr 12 11:05:56 2019][2937278.413559] LNet: Service thread pid 118364 was inactive for 200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 11:05:56 2019][2937278.430779] Pid: 118364, comm: mdt02_044 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:05:56 2019][2937278.440802] Call Trace: [Fri Apr 12 11:05:56 2019][2937278.443463] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:05:56 2019][2937278.450593] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:05:56 2019][2937278.457989] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:05:56 2019][2937278.465001] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:05:56 2019][2937278.472176] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:05:56 2019][2937278.479268] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:05:56 2019][2937278.486009] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:05:56 2019][2937278.492684] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:05:56 2019][2937278.499643] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:05:56 2019][2937278.506939] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:05:56 2019][2937278.513296] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:05:56 2019][2937278.520436] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:05:56 2019][2937278.528357] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:05:56 2019][2937278.534928] [] kthread+0xd1/0xe0 [Fri Apr 12 11:05:56 2019][2937278.540048] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:05:56 2019][2937278.546696] [] 0xffffffffffffffff [Fri Apr 12 11:05:56 2019][2937278.551914] LustreError: dumping log to /tmp/lustre-log.1555092356.118364 [Fri Apr 12 11:05:58 2019][2937280.039726] LNet: Service thread pid 118300 was inactive for 200.40s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 11:05:58 2019][2937280.056957] Pid: 118300, comm: mdt03_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:05:58 2019][2937280.066980] Call Trace: [Fri Apr 12 11:05:58 2019][2937280.069651] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.076782] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.084195] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:05:58 2019][2937280.090707] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:05:58 2019][2937280.097387] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:05:58 2019][2937280.103619] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:05:58 2019][2937280.110371] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:05:58 2019][2937280.116264] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.123433] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.131353] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.137861] [] kthread+0xd1/0xe0 [Fri Apr 12 11:05:58 2019][2937280.142949] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:05:58 2019][2937280.149640] [] 0xffffffffffffffff [Fri Apr 12 11:05:58 2019][2937280.154838] Pid: 118578, comm: mdt02_102 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:05:58 2019][2937280.164841] Call Trace: [Fri Apr 12 11:05:58 2019][2937280.167477] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.174603] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.182004] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:05:58 2019][2937280.189034] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:05:58 2019][2937280.196226] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:05:58 2019][2937280.203365] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:05:58 2019][2937280.210108] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:05:58 2019][2937280.216766] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.223707] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.230996] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.237349] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.244472] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.252396] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.258902] [] kthread+0xd1/0xe0 [Fri Apr 12 11:05:58 2019][2937280.263988] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:05:58 2019][2937280.270638] [] 0xffffffffffffffff [Fri Apr 12 11:05:58 2019][2937280.275835] Pid: 118213, comm: mdt00_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:05:58 2019][2937280.285840] Call Trace: [Fri Apr 12 11:05:58 2019][2937280.288477] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.295644] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.303045] [] mdt_reint_rename+0x13af/0x28e0 [mdt] [Fri Apr 12 11:05:58 2019][2937280.309812] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:05:58 2019][2937280.316056] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:05:58 2019][2937280.322838] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:05:58 2019][2937280.328740] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.335902] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.343832] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.350369] [] kthread+0xd1/0xe0 [Fri Apr 12 11:05:58 2019][2937280.355460] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:05:58 2019][2937280.362170] [] 0xffffffffffffffff [Fri Apr 12 11:05:58 2019][2937280.367375] Pid: 118327, comm: mdt01_067 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:05:58 2019][2937280.377388] Call Trace: [Fri Apr 12 11:05:58 2019][2937280.380031] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.387146] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.394530] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:05:58 2019][2937280.401532] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:05:58 2019][2937280.408719] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:05:58 2019][2937280.415810] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:05:58 2019][2937280.422565] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:05:58 2019][2937280.429224] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.436166] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.443472] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.449817] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.456957] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.464890] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:05:58 2019][2937280.471441] [] kthread+0xd1/0xe0 [Fri Apr 12 11:05:58 2019][2937280.476527] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:05:58 2019][2937280.483175] [] 0xffffffffffffffff [Fri Apr 12 11:05:58 2019][2937280.488392] LNet: Service thread pid 118568 was inactive for 202.44s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Fri Apr 12 11:05:58 2019][2937280.501529] LNet: Skipped 1 previous similar message [Fri Apr 12 11:06:00 2019][2937282.509647] LustreError: dumping log to /tmp/lustre-log.1555092360.117898 [Fri Apr 12 11:06:01 2019][2937283.533641] LustreError: dumping log to /tmp/lustre-log.1555092361.118397 [Fri Apr 12 11:06:07 2019][2937289.677770] LustreError: dumping log to /tmp/lustre-log.1555092367.117548 [Fri Apr 12 11:06:09 2019][2937291.725813] LustreError: dumping log to /tmp/lustre-log.1555092369.118337 [Fri Apr 12 11:06:11 2019][2937293.773861] LustreError: dumping log to /tmp/lustre-log.1555092371.118273 [Fri Apr 12 11:06:15 2019][2937296.879823] Lustre: fir-MDT0003: haven't heard from client 33efd1f4-7880-2df3-5781-2f84ef9a664e (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b5f3de68800, cur 1555092375 expire 1555092225 last 1555092148 [Fri Apr 12 11:06:15 2019][2937296.901759] Lustre: Skipped 99 previous similar messages [Fri Apr 12 11:06:18 2019][2937299.917983] LustreError: dumping log to /tmp/lustre-log.1555092378.118382 [Fri Apr 12 11:06:19 2019][2937300.877010] Lustre: fir-MDT0001: haven't heard from client 33efd1f4-7880-2df3-5781-2f84ef9a664e (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b50a2b9a800, cur 1555092379 expire 1555092229 last 1555092152 [Fri Apr 12 11:06:21 2019][2937302.991820] LustreError: dumping log to /tmp/lustre-log.1555092381.118157 [Fri Apr 12 11:06:23 2019][2937305.106220] LustreError: 119100:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b6a04653900 x1629297538766736/t0(0) o37->a77148c9-d434-623d-b422-9d417c28aa4b@10.8.1.32@o2ib6:23/0 lens 448/440 e 0 to 0 dl 1555092413 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:06:29 2019][2937311.694230] LustreError: dumping log to /tmp/lustre-log.1555092389.118468 [Fri Apr 12 11:06:30 2019][2937312.718244] LustreError: dumping log to /tmp/lustre-log.1555092390.118585 [Fri Apr 12 11:06:35 2019][2937317.326361] LustreError: dumping log to /tmp/lustre-log.1555092395.117466 [Fri Apr 12 11:06:37 2019][2937319.374403] LustreError: dumping log to /tmp/lustre-log.1555092397.118219 [Fri Apr 12 11:06:45 2019][2937327.566570] LustreError: dumping log to /tmp/lustre-log.1555092405.118749 [Fri Apr 12 11:06:48 2019][2937330.638827] LustreError: dumping log to /tmp/lustre-log.1555092408.118199 [Fri Apr 12 11:06:51 2019][2937332.951891] LustreError: 117532:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092321, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b7ca7ae9d40/0x53d0f223fc019eea lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 63 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 117532 timeout: 0 lvb_type: 0 [Fri Apr 12 11:06:51 2019][2937332.991470] LustreError: 117532:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 3 previous similar messages [Fri Apr 12 11:06:51 2019][2937333.080694] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Fri Apr 12 11:06:51 2019][2937333.096031] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Fri Apr 12 11:06:51 2019][2937333.106041] Lustre: Skipped 1 previous similar message [Fri Apr 12 11:07:03 2019][2937345.487964] LNet: Service thread pid 118255 was inactive for 200.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Fri Apr 12 11:07:03 2019][2937345.501084] LNet: Skipped 29 previous similar messages [Fri Apr 12 11:07:03 2019][2937345.506410] LustreError: dumping log to /tmp/lustre-log.1555092423.118255 [Fri Apr 12 11:07:08 2019][2937350.095067] LustreError: dumping log to /tmp/lustre-log.1555092428.118447 [Fri Apr 12 11:07:14 2019][2937356.239198] LustreError: dumping log to /tmp/lustre-log.1555092434.118321 [Fri Apr 12 11:07:21 2019][2937362.895334] LustreError: dumping log to /tmp/lustre-log.1555092441.118357 [Fri Apr 12 11:07:27 2019][2937369.381189] LustreError: 118848:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8b52e0668050 x1628593879168544/t0(0) o4->27f7bc82-09bc-7eda-09a0-c245e8cb614d@10.8.27.19@o2ib6:27/0 lens 488/448 e 0 to 0 dl 1555092477 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:07:27 2019][2937369.405746] Lustre: fir-MDT0002: Bulk IO write error with 27f7bc82-09bc-7eda-09a0-c245e8cb614d (at 10.8.27.19@o2ib6), client will retry: rc = -107 [Fri Apr 12 11:07:29 2019][2937371.599519] LustreError: dumping log to /tmp/lustre-log.1555092449.118264 [Fri Apr 12 11:07:33 2019][2937375.759602] Lustre: 118182:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:07:33 2019][2937375.759602] req@ffff8b619b71c850 x1629309860045184/t0(0) o36->72c3285d-d1f7-51c7-3c9c-459f7c3fc02c@10.9.101.60@o2ib4:8/0 lens 512/2888 e 0 to 0 dl 1555092458 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:07:33 2019][2937375.788813] Lustre: 118182:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 32 previous similar messages [Fri Apr 12 11:07:43 2019][2937384.912787] LustreError: dumping log to /tmp/lustre-log.1555092463.118507 [Fri Apr 12 11:07:46 2019][2937387.983843] LustreError: dumping log to /tmp/lustre-log.1555092466.118245 [Fri Apr 12 11:07:48 2019][2937390.032886] LustreError: dumping log to /tmp/lustre-log.1555092468.117551 [Fri Apr 12 11:07:48 2019][2937390.413463] LustreError: 117517:0:(mdt_io.c:442:mdt_preprw_write()) fir-MDT0002: BRW to missing obj [0x2c001122a:0xa471:0x0] [Fri Apr 12 11:08:01 2019][2937403.344153] LustreError: dumping log to /tmp/lustre-log.1555092481.118465 [Fri Apr 12 11:08:06 2019][2937408.464301] LustreError: dumping log to /tmp/lustre-log.1555092486.118414 [Fri Apr 12 11:08:09 2019][2937411.179973] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) [Fri Apr 12 11:08:09 2019][2937411.192954] LNetError: 117117:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 10 previous similar messages [Fri Apr 12 11:08:15 2019][2937417.680644] LustreError: dumping log to /tmp/lustre-log.1555092495.117968 [Fri Apr 12 11:08:16 2019][2937418.192451] LustreError: dumping log to /tmp/lustre-log.1555092496.118431 [Fri Apr 12 11:08:17 2019][2937419.216477] LustreError: dumping log to /tmp/lustre-log.1555092497.117491 [Fri Apr 12 11:08:21 2019][2937423.312555] LustreError: dumping log to /tmp/lustre-log.1555092501.118294 [Fri Apr 12 11:08:22 2019][2937424.336576] LustreError: dumping log to /tmp/lustre-log.1555092502.118399 [Fri Apr 12 11:08:23 2019][2937425.360623] LustreError: dumping log to /tmp/lustre-log.1555092503.118351 [Fri Apr 12 11:08:30 2019][2937432.016732] LustreError: dumping log to /tmp/lustre-log.1555092510.118483 [Fri Apr 12 11:08:32 2019][2937434.064860] LustreError: dumping log to /tmp/lustre-log.1555092512.118557 [Fri Apr 12 11:08:41 2019][2937443.280957] LustreError: dumping log to /tmp/lustre-log.1555092521.118694 [Fri Apr 12 11:08:42 2019][2937444.265334] Lustre: fir-MDT0000: Connection restored to f51798bd-42f7-774a-f83c-6082c498f2ba (at 10.8.18.10@o2ib6) [Fri Apr 12 11:08:42 2019][2937444.275885] Lustre: Skipped 2428 previous similar messages [Fri Apr 12 11:08:42 2019][2937444.305973] LustreError: dumping log to /tmp/lustre-log.1555092522.117869 [Fri Apr 12 11:08:43 2019][2937445.713993] Lustre: fir-MDT0000-osp-MDT0003: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Fri Apr 12 11:08:43 2019][2937445.729977] Lustre: Skipped 1 previous similar message [Fri Apr 12 11:08:43 2019][2937445.736980] LustreError: 118329:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092433, 90s ago), entering recovery for fir-MDT0000_UUID@10.0.10.51@o2ib7 ns: fir-MDT0000-osp-MDT0003 lock: ffff8b65c7f32f40/0x53d0f22400bbe333 lrc: 4/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x1000001000000 nid: local remote: 0x53d0f22400bbe372 expref: -99 pid: 118329 timeout: 0 lvb_type: 0 [Fri Apr 12 11:08:43 2019][2937445.777899] LustreError: 118329:0:(ldlm_request.c:147:ldlm_expired_completion_wait()) Skipped 28 previous similar messages [Fri Apr 12 11:08:44 2019][2937446.354006] LustreError: dumping log to /tmp/lustre-log.1555092524.118358 [Fri Apr 12 11:08:53 2019][2937455.057171] LustreError: dumping log to /tmp/lustre-log.1555092533.118267 [Fri Apr 12 11:08:54 2019][2937456.081197] LustreError: dumping log to /tmp/lustre-log.1555092534.118366 [Fri Apr 12 11:08:54 2019][2937456.492399] Lustre: fir-MDT0002: Client ef2a72e8-1102-fad1-812c-2f0167c49e1f (at 10.9.101.67@o2ib4) reconnecting [Fri Apr 12 11:08:54 2019][2937456.502773] Lustre: Skipped 2442 previous similar messages [Fri Apr 12 11:08:59 2019][2937461.713291] LustreError: dumping log to /tmp/lustre-log.1555092539.118324 [Fri Apr 12 11:09:05 2019][2937467.857408] LustreError: dumping log to /tmp/lustre-log.1555092545.118226 [Fri Apr 12 11:09:07 2019][2937469.655447] LustreError: 117550:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092457, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b57c9a8a1c0/0x53d0f22401d6257e lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 79 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 117550 timeout: 0 lvb_type: 0 [Fri Apr 12 11:09:07 2019][2937469.695019] LustreError: 117550:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 7 previous similar messages [Fri Apr 12 11:09:08 2019][2937469.905452] LustreError: dumping log to /tmp/lustre-log.1555092548.118380 [Fri Apr 12 11:09:19 2019][2937481.047889] Lustre: fir-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [Fri Apr 12 11:09:19 2019][2937481.057914] Lustre: Skipped 2 previous similar messages [Fri Apr 12 11:09:22 2019][2937484.241740] LNet: Service thread pid 118335 was inactive for 200.28s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Fri Apr 12 11:09:22 2019][2937484.254898] LNet: Skipped 37 previous similar messages [Fri Apr 12 11:09:22 2019][2937484.260257] LustreError: dumping log to /tmp/lustre-log.1555092562.118335 [Fri Apr 12 11:09:37 2019][2937499.602042] LustreError: dumping log to /tmp/lustre-log.1555092577.118355 [Fri Apr 12 11:09:44 2019][2937506.770189] LustreError: dumping log to /tmp/lustre-log.1555092584.118326 [Fri Apr 12 11:09:51 2019][2937512.914335] LustreError: dumping log to /tmp/lustre-log.1555092591.118724 [Fri Apr 12 11:10:12 2019][2937534.419742] LustreError: dumping log to /tmp/lustre-log.1555092612.118526 [Fri Apr 12 11:10:13 2019][2937535.108010] SysRq : Show State [Fri Apr 12 11:10:13 2019][2937535.111282] task PC stack pid father [Fri Apr 12 11:10:13 2019][2937535.117403] systemd S ffff8b5da9eb8000 0 1 0 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.124713] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.127352] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.132505] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:13 2019][2937535.139737] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Fri Apr 12 11:10:13 2019][2937535.146708] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:13 2019][2937535.153245] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:13 2019][2937535.158487] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:13 2019][2937535.163636] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:13 2019][2937535.169394] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:13 2019][2937535.175155] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:13 2019][2937535.181343] kthreadd S ffff8b5da9eb9040 0 2 0 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.188617] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.191248] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.196396] [] kthreadd+0x2f5/0x300 [Fri Apr 12 11:10:13 2019][2937535.201719] [] ? kthread_create_on_cpu+0x60/0x60 [Fri Apr 12 11:10:13 2019][2937535.208167] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.214803] [] ? kthread_create_on_cpu+0x60/0x60 [Fri Apr 12 11:10:13 2019][2937535.221271] ksoftirqd/0 S ffff8b4e29b80000 0 3 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.228560] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.231196] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.236346] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:13 2019][2937535.242366] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.248382] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.253444] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.259720] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.266356] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.272646] kworker/0:0H S ffff8b4e29b82080 0 5 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.279937] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.282575] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:13 2019][2937535.288767] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.293915] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:13 2019][2937535.299672] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:13 2019][2937535.306377] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.311441] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.317716] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.324355] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.330625] migration/0 S ffff8b4e29b85140 0 8 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.337946] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.340579] [] ? cpu_stop_queue_work+0x70/0x70 [Fri Apr 12 11:10:13 2019][2937535.346853] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.351999] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:13 2019][2937535.358022] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.364039] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.369106] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.375381] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.382004] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.388279] rcu_bh S ffff8b4e29b86180 0 9 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.395554] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.398187] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:13 2019][2937535.403853] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.409002] [] rcu_gp_kthread+0xb2/0x710 [Fri Apr 12 11:10:13 2019][2937535.414756] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:13 2019][2937535.420770] [] ? rcu_gp_fqs+0x90/0x90 [Fri Apr 12 11:10:13 2019][2937535.426263] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.431326] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.437602] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.444228] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.450521] rcu_sched S ffff8b4e29ba8000 0 10 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.457864] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.460514] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.465676] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:13 2019][2937535.471692] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:13 2019][2937535.478244] [] ? prepare_to_wait+0x56/0x90 [Fri Apr 12 11:10:13 2019][2937535.484188] [] rcu_gp_kthread+0x349/0x710 [Fri Apr 12 11:10:13 2019][2937535.490028] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:13 2019][2937535.496045] [] ? rcu_gp_fqs+0x90/0x90 [Fri Apr 12 11:10:13 2019][2937535.501536] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.506609] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.512890] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.519515] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.525821] lru-add-drain S ffff8b4e29ba9040 0 11 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.533114] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.535749] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.541021] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:13 2019][2937535.546891] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:13 2019][2937535.553084] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.558143] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.564426] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.571046] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.577322] watchdog/0 S ffff8b4e29baa080 0 12 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.584596] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.587227] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:13 2019][2937535.592722] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.597880] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:13 2019][2937535.603903] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.609917] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.615008] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.621279] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.627899] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.634174] watchdog/1 S ffff8b5da9f8b0c0 0 13 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.641655] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.644405] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:13 2019][2937535.649896] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.655049] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:13 2019][2937535.661073] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.667082] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.672142] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.678419] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.685037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.691306] migration/1 S ffff8b5da9fb0000 0 14 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.698590] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.701230] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:13 2019][2937535.707852] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.713003] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:13 2019][2937535.719024] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.725038] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.730098] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.736379] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.743009] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.749306] ksoftirqd/1 S ffff8b5da9fb1040 0 15 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.756582] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.759213] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.764359] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:13 2019][2937535.770372] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.776386] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.781440] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.787715] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.794336] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.800610] kworker/1:0H S ffff8b5da9fb30c0 0 17 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.807887] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.810521] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:13 2019][2937535.816706] [] schedule+0x29/0x70 [Fri Apr 12 11:10:13 2019][2937535.821854] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:13 2019][2937535.827623] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:13 2019][2937535.834365] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:13 2019][2937535.839425] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.845708] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:13 2019][2937535.852329] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:13 2019][2937535.858601] watchdog/2 S ffff8b6da981a080 0 19 2 0x00000000 [Fri Apr 12 11:10:13 2019][2937535.865886] Call Trace: [Fri Apr 12 11:10:13 2019][2937535.868520] [] ? watchdog_cleanup+0x10/0x10 [Fri Apr 12 11:10:14 2019][2937535.874535] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:14 2019][2937535.880027] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937535.885174] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937535.891208] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937535.897224] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937535.902286] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937535.908560] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937535.915194] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937535.921463] migration/2 S ffff8b6da981b0c0 0 20 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937535.928739] Call Trace: [Fri Apr 12 11:10:14 2019][2937535.931371] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:14 2019][2937535.938010] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937535.943157] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937535.949168] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937535.955178] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937535.960237] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937535.966511] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937535.973126] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937535.979399] ksoftirqd/2 S ffff8b6da981c100 0 21 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937535.986697] Call Trace: [Fri Apr 12 11:10:14 2019][2937535.989337] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937535.994484] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.000497] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.006513] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.011573] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.017849] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.024470] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.030745] kworker/2:0H S ffff8b6da9841040 0 23 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.038055] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.040689] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:14 2019][2937536.046876] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.052026] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:14 2019][2937536.057793] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:14 2019][2937536.064508] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.069585] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.075859] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.082477] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.088753] watchdog/3 S ffff8b7da94c2080 0 25 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.096026] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.098660] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:14 2019][2937536.104155] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.109307] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.115358] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.121376] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.126466] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.132741] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.139388] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.145659] migration/3 S ffff8b7da94c30c0 0 26 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.152943] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.155578] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:14 2019][2937536.162259] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.167419] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.173434] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.179451] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.184515] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.190790] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.197410] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.203684] ksoftirqd/3 S ffff8b7da94c4100 0 27 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.211006] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.213648] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.218801] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.224814] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.230828] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.235886] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.242158] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.248776] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.255082] kworker/3:0H S ffff8b7da94e1040 0 29 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.262369] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.265005] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:14 2019][2937536.271191] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.276339] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:14 2019][2937536.282093] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:14 2019][2937536.288798] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.293864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.300137] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.306758] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.313033] watchdog/4 S ffff8b4e294830c0 0 31 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.320307] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.322941] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:14 2019][2937536.328436] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.333583] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.339602] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.345623] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.350685] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.356988] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.363617] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.369900] migration/4 S ffff8b4e294d0000 0 32 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.377261] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.379915] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:14 2019][2937536.386540] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.391685] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.397699] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.403725] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.408806] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.415077] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.421699] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.427972] ksoftirqd/4 S ffff8b4e294d1040 0 33 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.435247] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.437880] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.443030] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.449042] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.455059] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.460120] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.466394] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.473015] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.479290] kworker/4:0H S ffff8b4e294d30c0 0 35 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.486570] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.489207] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:14 2019][2937536.495396] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.500541] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:14 2019][2937536.506297] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:14 2019][2937536.513004] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.518065] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.524340] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.530963] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.537236] watchdog/5 S ffff8b5da9fb6180 0 36 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.544512] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.547143] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:14 2019][2937536.552639] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.557784] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.563802] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.569815] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.574876] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.581152] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.587777] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.594046] migration/5 S ffff8b5da9fb5140 0 37 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.601322] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.603956] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.609103] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.615142] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.621156] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.626216] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.632487] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.639106] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.645383] ksoftirqd/5 S ffff8b5da9ff0000 0 38 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.652655] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.655289] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.660436] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.666452] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.672465] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.677524] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.683803] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.690423] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.696696] kworker/5:0H S ffff8b5da9ff2080 0 40 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.703980] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.706615] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:14 2019][2937536.712802] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.717949] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:14 2019][2937536.723706] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:14 2019][2937536.730411] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.735475] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.741747] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.748371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.754644] watchdog/6 S ffff8b6da9846180 0 41 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.761927] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.764560] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:14 2019][2937536.770056] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.775210] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.781224] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.787241] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.792301] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.798578] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.805197] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.811474] migration/6 S ffff8b6da9845140 0 42 2 0x00000000 [Fri Apr 12 11:10:14 2019][2937536.818747] Call Trace: [Fri Apr 12 11:10:14 2019][2937536.821381] [] ? cpu_stop_queue_work+0x70/0x70 [Fri Apr 12 11:10:14 2019][2937536.827655] [] schedule+0x29/0x70 [Fri Apr 12 11:10:14 2019][2937536.832803] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:14 2019][2937536.838819] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.844833] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:14 2019][2937536.849895] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.856169] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:14 2019][2937536.862791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:14 2019][2937536.869063] ksoftirqd/6 S ffff8b6da9844100 0 43 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937536.876338] Call Trace: [Fri Apr 12 11:10:15 2019][2937536.878973] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937536.884121] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937536.890135] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937536.896151] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937536.901211] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937536.907489] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937536.914108] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937536.920382] kworker/6:0H S ffff8b6da9888000 0 45 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937536.927663] Call Trace: [Fri Apr 12 11:10:15 2019][2937536.930297] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:15 2019][2937536.936488] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937536.941634] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:15 2019][2937536.947388] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:15 2019][2937536.954093] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937536.959158] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937536.965431] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937536.972053] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937536.978327] watchdog/7 S ffff8b7da94e6180 0 46 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937536.985603] Call Trace: [Fri Apr 12 11:10:15 2019][2937536.988234] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:15 2019][2937536.993731] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937536.998878] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.004891] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.010905] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.015968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.022244] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.028866] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.035136] migration/7 S ffff8b7da94e5140 0 47 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.042406] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.045039] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.050185] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.056201] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.062215] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.067274] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.073552] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.080172] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.086445] ksoftirqd/7 S ffff8b7da94e4100 0 48 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.093730] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.096363] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.101511] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.107526] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.113539] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.118600] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.124876] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.131496] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.137771] kworker/7:0H S ffff8b7da9520000 0 50 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.145052] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.147690] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:15 2019][2937537.153878] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.159025] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:15 2019][2937537.164780] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:15 2019][2937537.171500] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.176556] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.182834] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.189452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.195727] watchdog/8 S ffff8b4e294d6180 0 51 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.203002] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.205635] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:15 2019][2937537.211129] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.216278] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.222294] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.228305] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.233367] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.239646] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.246267] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.252537] migration/8 S ffff8b4e294d5140 0 52 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.259813] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.262444] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.267594] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.273606] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.279622] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.284684] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.290961] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.297580] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.303854] ksoftirqd/8 S ffff8b4e294d4100 0 53 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.311129] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.313763] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.318911] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.324923] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.330939] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.336000] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.342275] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.348896] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.355172] kworker/8:0H S ffff8b4e29519040 0 55 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.362454] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.365088] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:15 2019][2937537.371276] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.376422] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:15 2019][2937537.382179] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:15 2019][2937537.388883] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.393949] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.400223] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.406847] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.413119] watchdog/9 S ffff8b5da9ff6180 0 56 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.420392] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.423025] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:15 2019][2937537.428520] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.433667] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.439684] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.445699] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.450761] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.457034] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.463675] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.469957] migration/9 S ffff8b5da9ff5140 0 57 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.477241] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.479882] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:15 2019][2937537.486531] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.491677] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.497700] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.503722] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.508783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.515058] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.521681] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.527954] ksoftirqd/9 S ffff8b5da9ff4100 0 58 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.535247] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.537877] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.543025] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.549038] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.555055] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.560116] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.566391] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.573012] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.579287] kworker/9:0H S ffff8b5da9820000 0 60 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.586569] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.589202] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:15 2019][2937537.595389] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.600540] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:15 2019][2937537.606293] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:15 2019][2937537.613000] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.618061] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.624338] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.630958] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.637231] watchdog/10 S ffff8b6da988e180 0 61 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.644499] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.647132] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:15 2019][2937537.653319] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:15 2019][2937537.658813] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.663961] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.669977] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.675992] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.681054] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.687332] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.693949] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.700222] migration/10 S ffff8b6da988d140 0 62 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.707492] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.710121] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:15 2019][2937537.716743] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.721891] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.727906] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.733921] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.738983] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.745257] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.751879] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.758151] ksoftirqd/10 S ffff8b6da988c100 0 63 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.765436] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.768069] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.773216] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:15 2019][2937537.779233] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.785246] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.790306] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.796582] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:15 2019][2937537.803204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:15 2019][2937537.809477] kworker/10:0H S ffff8b6da988a080 0 65 2 0x00000000 [Fri Apr 12 11:10:15 2019][2937537.816760] Call Trace: [Fri Apr 12 11:10:15 2019][2937537.819397] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:15 2019][2937537.825582] [] schedule+0x29/0x70 [Fri Apr 12 11:10:15 2019][2937537.830728] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:15 2019][2937537.836479] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:15 2019][2937537.843183] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:15 2019][2937537.848245] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937537.854520] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937537.861141] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937537.867414] watchdog/11 S ffff8b7da9526180 0 66 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937537.874684] Call Trace: [Fri Apr 12 11:10:16 2019][2937537.877316] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:16 2019][2937537.882809] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937537.887958] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937537.893972] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937537.899986] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937537.905046] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937537.911314] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937537.917938] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937537.924207] migration/11 S ffff8b7da9525140 0 67 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937537.931479] Call Trace: [Fri Apr 12 11:10:16 2019][2937537.934109] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:16 2019][2937537.940728] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937537.945870] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937537.951882] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937537.957890] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937537.962951] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937537.969226] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937537.975846] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937537.982110] ksoftirqd/11 S ffff8b7da9524100 0 68 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937537.989381] Call Trace: [Fri Apr 12 11:10:16 2019][2937537.992014] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937537.997161] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.003175] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.009187] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.014241] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.020517] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.027136] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.033403] kworker/11:0H S ffff8b7da9522080 0 70 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.040677] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.043311] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:16 2019][2937538.049500] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.054646] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:16 2019][2937538.060402] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:16 2019][2937538.067109] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.072169] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.078437] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.085058] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.091333] watchdog/12 S ffff8b4e2951e180 0 71 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.098610] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.101240] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:16 2019][2937538.106736] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.111882] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.117895] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.123903] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.128966] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.135238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.141853] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.148125] migration/12 S ffff8b4e2951d140 0 72 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.155396] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.158027] [] ? cpu_stop_queue_work+0x70/0x70 [Fri Apr 12 11:10:16 2019][2937538.164300] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:16 2019][2937538.170921] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.176070] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.182083] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.188099] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.193159] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.199434] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.206047] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.212323] ksoftirqd/12 S ffff8b4e2951c100 0 73 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.219600] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.222231] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.227377] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.233390] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.239396] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.244453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.250725] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.257347] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.263622] kworker/12:0H S ffff8b4e2951a080 0 75 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.270903] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.273537] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:16 2019][2937538.279727] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.284874] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:16 2019][2937538.290628] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:16 2019][2937538.297336] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.302398] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.308672] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.315292] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.321558] watchdog/13 S ffff8b5da9826180 0 76 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.328829] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.331458] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:16 2019][2937538.336953] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.342101] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.348115] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.354131] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.359190] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.365467] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.372087] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.378360] migration/13 S ffff8b5da9825140 0 77 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.385629] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.388260] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:16 2019][2937538.394883] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.400030] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.406043] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.412060] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.417120] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.423397] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.430017] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.436291] ksoftirqd/13 S ffff8b5da9824100 0 78 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.443559] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.446189] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.451338] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.457352] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.463365] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.468420] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.474696] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.481317] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.487592] kworker/13:0H S ffff8b5da9822080 0 80 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.494874] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.497509] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:16 2019][2937538.503693] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.508835] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:16 2019][2937538.514587] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:16 2019][2937538.521289] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.526348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.532623] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.539246] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.545519] watchdog/14 S ffff8b6da9889040 0 81 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.552787] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.555421] [] ? watchdog_cleanup+0x10/0x10 [Fri Apr 12 11:10:16 2019][2937538.561431] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:16 2019][2937538.566937] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.572086] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.578098] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.584105] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.589166] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.595443] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.602064] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.608337] migration/14 S ffff8b6da98d0000 0 82 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.615614] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.618247] [] ? cpu_stop_queue_work+0x70/0x70 [Fri Apr 12 11:10:16 2019][2937538.624520] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.629670] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.635684] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.641696] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.646759] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.653033] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.659654] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.665927] ksoftirqd/14 S ffff8b6da98d1040 0 83 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.673198] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.675827] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.680974] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.686990] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.693004] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.698065] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.704341] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.710971] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.717247] kworker/14:0H S ffff8b6da98d30c0 0 85 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.724537] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.727173] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:16 2019][2937538.733360] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.738508] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:16 2019][2937538.744262] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:16 2019][2937538.750968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.756030] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.762305] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.768928] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.775199] watchdog/15 S ffff8b7da9521040 0 86 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.782468] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.785100] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:16 2019][2937538.790595] [] schedule+0x29/0x70 [Fri Apr 12 11:10:16 2019][2937538.795742] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:16 2019][2937538.801757] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.807772] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:16 2019][2937538.812833] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.819110] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:16 2019][2937538.825728] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:16 2019][2937538.832002] migration/15 R running task 0 87 2 0x00000000 [Fri Apr 12 11:10:16 2019][2937538.839271] Call Trace: [Fri Apr 12 11:10:16 2019][2937538.841902] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:16 2019][2937538.848524] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937538.853671] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937538.859687] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937538.865699] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937538.870760] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937538.877029] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937538.883650] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937538.889922] ksoftirqd/15 R running task 0 88 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937538.897192] Call Trace: [Fri Apr 12 11:10:17 2019][2937538.899826] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937538.904968] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937538.910975] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937538.916981] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937538.922042] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937538.928310] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937538.934932] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937538.941206] kworker/15:0H S ffff8b7da956b0c0 0 90 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937538.948488] Call Trace: [Fri Apr 12 11:10:17 2019][2937538.951121] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:17 2019][2937538.957310] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937538.962458] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:17 2019][2937538.968213] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:17 2019][2937538.974920] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937538.979982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937538.986256] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937538.992880] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937538.999152] watchdog/16 S ffff8b4e295b0000 0 91 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.006431] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.009058] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:17 2019][2937539.014554] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.019700] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.025707] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.031724] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.036785] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.043061] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.049680] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.055952] migration/16 S ffff8b4e295b1040 0 92 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.063223] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.065855] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:17 2019][2937539.072476] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.077624] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.083638] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.089650] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.094704] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.100970] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.107593] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.113868] ksoftirqd/16 S ffff8b4e295b2080 0 93 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.121145] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.123773] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.128922] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.134937] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.140953] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.146012] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.152289] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.158907] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.165174] kworker/16:0H S ffff8b4e295b4100 0 95 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.172460] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.175093] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:17 2019][2937539.181278] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.186427] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:17 2019][2937539.192182] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:17 2019][2937539.198891] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.203948] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.210219] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.216838] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.223111] watchdog/17 S ffff8b5da9821040 0 96 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.230380] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.233011] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:17 2019][2937539.238506] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.243654] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.249668] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.255684] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.260746] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.267021] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.273642] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.279915] migration/17 S ffff8b5da9868000 0 97 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.287191] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.289824] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.294969] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.300977] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.306991] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.312051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.318318] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.324941] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.331215] ksoftirqd/17 S ffff8b5da9869040 0 98 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.338490] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.341123] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.346268] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.352275] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.358290] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.363353] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.369645] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.376267] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.382540] kworker/17:0H S ffff8b5da986b0c0 0 100 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.389821] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.392457] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:17 2019][2937539.398645] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.403792] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:17 2019][2937539.409546] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:17 2019][2937539.416254] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.421314] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.427597] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.434213] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.440485] watchdog/18 S ffff8b6da98d6180 0 101 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.447754] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.450385] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:17 2019][2937539.455881] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.461029] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.467041] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.473058] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.478119] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.484393] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.491016] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.497287] migration/18 S ffff8b6da98d5140 0 102 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.504558] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.507191] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:17 2019][2937539.513810] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.518957] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.524972] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.530988] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.536047] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.542323] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.548944] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.555218] ksoftirqd/18 S ffff8b6da98d4100 0 103 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.562503] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.565138] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.570282] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.576296] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.582311] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.587371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.593647] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.600284] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.606551] kworker/18:0H S ffff8b6da9921040 0 105 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.613825] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.616461] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:17 2019][2937539.622647] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.627796] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:17 2019][2937539.633550] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:17 2019][2937539.640259] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.645324] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.651597] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.658216] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.664489] watchdog/19 S ffff8b7da956e180 0 106 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.671759] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.674390] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:17 2019][2937539.679883] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.685032] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.691043] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.697050] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.702106] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.708380] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.715000] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.721276] migration/19 S ffff8b7da956d140 0 107 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.728552] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.731183] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.736329] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.742336] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.748351] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.753412] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.759686] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.766299] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.772573] ksoftirqd/19 S ffff8b7da956c100 0 108 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.779841] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.782474] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.787621] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:17 2019][2937539.793635] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.799652] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:17 2019][2937539.804710] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.810977] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:17 2019][2937539.817598] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:17 2019][2937539.823864] kworker/19:0H S ffff8b7da95b9040 0 110 2 0x00000000 [Fri Apr 12 11:10:17 2019][2937539.831149] Call Trace: [Fri Apr 12 11:10:17 2019][2937539.833780] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:17 2019][2937539.839967] [] schedule+0x29/0x70 [Fri Apr 12 11:10:17 2019][2937539.845107] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:17 2019][2937539.850864] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:18 2019][2937539.857571] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937539.862635] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937539.868907] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937539.875530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937539.881802] watchdog/20 S ffff8b4e295b6180 0 111 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937539.889088] Call Trace: [Fri Apr 12 11:10:18 2019][2937539.891721] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:18 2019][2937539.897214] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937539.902361] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937539.908375] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937539.914391] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937539.919452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937539.925729] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937539.932350] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937539.938622] migration/20 S ffff8b4e295b5140 0 112 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937539.945891] Call Trace: [Fri Apr 12 11:10:18 2019][2937539.948524] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:18 2019][2937539.955144] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937539.960291] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937539.966304] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937539.972320] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937539.977382] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937539.983658] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937539.990278] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937539.996551] ksoftirqd/20 S ffff8b4e29608000 0 113 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.003819] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.006450] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.011600] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.017613] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.023628] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.028690] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.034964] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.041586] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.047860] kworker/20:0H S ffff8b4e2960a080 0 115 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.055136] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.057770] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:18 2019][2937540.063957] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.069103] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:18 2019][2937540.074858] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:18 2019][2937540.081565] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.086619] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.092895] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.099516] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.105788] watchdog/21 S ffff8b5da986e180 0 116 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.113058] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.115687] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:18 2019][2937540.121181] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.126331] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.132345] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.138360] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.143422] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.149697] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.156319] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.162592] migration/21 S ffff8b5da986d140 0 117 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.169868] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.172501] [] ? cpu_stop_queue_work+0x70/0x70 [Fri Apr 12 11:10:18 2019][2937540.178774] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:18 2019][2937540.185395] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.190543] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.196556] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.202564] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.207628] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.213902] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.220521] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.226796] ksoftirqd/21 S ffff8b5da986c100 0 118 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.234090] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.236720] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.241869] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.247882] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.253887] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.258940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.265207] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.271829] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.278105] kworker/21:0H S ffff8b5da98b9040 0 120 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.285385] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.288021] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:18 2019][2937540.294208] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.299356] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:18 2019][2937540.305110] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:18 2019][2937540.311817] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.316873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.323145] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.329769] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.336042] watchdog/22 S ffff8b6da9926180 0 121 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.343320] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.345949] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:18 2019][2937540.351444] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.356594] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.362607] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.368620] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.373674] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.379949] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.386570] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.392848] migration/22 S ffff8b6da9925140 0 122 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.400121] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.402753] [] ? cpu_stop_queue_work+0x70/0x70 [Fri Apr 12 11:10:18 2019][2937540.409027] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.414176] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.420189] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.426203] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.431256] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.437533] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.444153] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.450428] ksoftirqd/22 S ffff8b6da9924100 0 123 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.457704] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.460334] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.465483] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.471494] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.477501] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.482555] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.488832] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.495451] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.501725] kworker/22:0H S ffff8b6da9922080 0 125 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.509000] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.511634] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:18 2019][2937540.517822] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.522969] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:18 2019][2937540.528725] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:18 2019][2937540.535433] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.540494] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.546768] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.553388] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.559654] watchdog/23 S ffff8b7da95be180 0 126 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.566925] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.569552] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:18 2019][2937540.575041] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.580188] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.586201] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.592208] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.597262] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.603536] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.610158] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.616432] migration/23 S ffff8b7da95bd140 0 127 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.623709] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.626338] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.631487] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.637500] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.643508] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.648569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.654843] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.661466] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.667738] ksoftirqd/23 S ffff8b7da95bc100 0 128 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.675008] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.677637] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.682785] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.688800] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.694807] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.699869] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.706144] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.712765] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.719039] kworker/23:0H S ffff8b7da95ba080 0 130 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.726314] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.728947] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:18 2019][2937540.735133] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.740274] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:18 2019][2937540.746029] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:18 2019][2937540.752736] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.757797] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.764066] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.770686] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.776958] watchdog/24 S ffff8b4e2960e180 0 131 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.784228] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.786861] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:18 2019][2937540.792356] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.797501] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.803515] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.809531] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:18 2019][2937540.814591] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.820870] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:18 2019][2937540.827488] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:18 2019][2937540.833761] migration/24 S ffff8b4e2960d140 0 132 2 0x00000000 [Fri Apr 12 11:10:18 2019][2937540.841032] Call Trace: [Fri Apr 12 11:10:18 2019][2937540.843664] [] schedule+0x29/0x70 [Fri Apr 12 11:10:18 2019][2937540.848809] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:18 2019][2937540.854823] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937540.860829] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937540.865890] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937540.872158] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937540.878779] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937540.885054] ksoftirqd/24 S ffff8b4e2960c100 0 133 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937540.892331] Call Trace: [Fri Apr 12 11:10:19 2019][2937540.894960] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937540.900109] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937540.906121] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937540.912127] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937540.917180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937540.923449] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937540.930070] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937540.936343] kworker/24:0H S ffff8b4e29658000 0 135 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937540.943625] Call Trace: [Fri Apr 12 11:10:19 2019][2937540.946261] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:19 2019][2937540.952447] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937540.957597] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:19 2019][2937540.963352] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:19 2019][2937540.970060] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937540.975118] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937540.981387] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937540.988006] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937540.994273] watchdog/25 S ffff8b5da98be180 0 136 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.001550] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.004180] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:19 2019][2937541.009675] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.014823] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.020838] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.026852] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.031913] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.038190] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.044811] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.051082] migration/25 S ffff8b5da98bd140 0 137 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.058351] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.060983] [] ? cpu_stop_unpark+0x50/0x50 [Fri Apr 12 11:10:19 2019][2937541.066911] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.072057] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.078065] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.084079] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.089158] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.095432] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.102056] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.108329] ksoftirqd/25 S ffff8b5da98bc100 0 138 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.115605] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.118237] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.123383] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.129390] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.135404] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.140465] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.146754] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.153371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.159645] kworker/25:0H S ffff8b5da98ba080 0 140 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.166928] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.169561] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:19 2019][2937541.175749] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.180896] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:19 2019][2937541.186644] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:19 2019][2937541.193351] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.198412] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.204702] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.211318] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.217592] watchdog/26 S ffff8b6da9970000 0 141 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.224860] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.227491] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:19 2019][2937541.233679] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:19 2019][2937541.239172] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.244313] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.250328] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.256342] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.261403] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.267679] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.274301] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.280573] migration/26 S ffff8b6da9971040 0 142 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.287843] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.290473] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.295619] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.301626] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.307640] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.312694] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.318971] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.325589] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.331865] ksoftirqd/26 S ffff8b6da9972080 0 143 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.339142] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.341773] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.346920] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.352935] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.358951] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.364011] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.370286] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.376908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.383180] kworker/26:0H S ffff8b6da9974100 0 145 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.390454] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.393089] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:19 2019][2937541.399276] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.404426] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:19 2019][2937541.410178] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:19 2019][2937541.416878] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.421938] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.428205] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.434827] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.441101] watchdog/27 S ffff8b7da9600000 0 146 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.448371] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.451000] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:19 2019][2937541.456494] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.461642] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.467655] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.473664] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.478724] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.484990] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.491603] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.497874] migration/27 S ffff8b7da9601040 0 147 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.505147] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.507778] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.512925] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.518939] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.524953] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.530005] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.536273] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.542896] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.549185] ksoftirqd/27 S ffff8b7da9602080 0 148 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.556454] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.559085] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.564232] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.570249] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.576263] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.581322] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.587589] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.594204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.600476] kworker/27:0H S ffff8b7da9604100 0 150 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.607750] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.610384] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:19 2019][2937541.616572] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.621720] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:19 2019][2937541.627477] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:19 2019][2937541.634183] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.639244] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.645520] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.652140] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.658415] watchdog/28 S ffff8b4e2965e180 0 151 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.665692] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.668322] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:19 2019][2937541.674508] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:19 2019][2937541.679997] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.685143] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.691159] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.697175] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.702236] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.708509] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.715132] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.721405] migration/28 S ffff8b4e2965d140 0 152 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.728673] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.731305] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:19 2019][2937541.737924] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.743074] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.749087] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.755095] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.760157] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.766432] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.773051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.779334] ksoftirqd/28 S ffff8b4e2965c100 0 153 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.786612] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.789242] [] schedule+0x29/0x70 [Fri Apr 12 11:10:19 2019][2937541.794388] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:19 2019][2937541.800396] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.806410] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:19 2019][2937541.811471] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.817748] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:19 2019][2937541.824370] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:19 2019][2937541.830641] kworker/28:0H S ffff8b4e2965a080 0 155 2 0x00000000 [Fri Apr 12 11:10:19 2019][2937541.837916] Call Trace: [Fri Apr 12 11:10:19 2019][2937541.840553] [] ? start_worker+0x27/0x30 [Fri Apr 12 11:10:19 2019][2937541.846221] [] ? manage_workers.isra.25+0x189/0x2a0 [Fri Apr 12 11:10:19 2019][2937541.852927] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937541.858075] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:20 2019][2937541.863829] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:20 2019][2937541.870539] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937541.875599] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937541.881876] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937541.888494] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937541.894768] watchdog/29 S ffff8b5da98f8000 0 156 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937541.902037] Call Trace: [Fri Apr 12 11:10:20 2019][2937541.904667] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:20 2019][2937541.910161] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937541.915310] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937541.921324] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937541.927339] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937541.932401] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937541.938677] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937541.945299] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937541.951587] migration/29 S ffff8b5da98f9040 0 157 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937541.958857] Call Trace: [Fri Apr 12 11:10:20 2019][2937541.961489] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937541.966635] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937541.972649] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937541.978664] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937541.983726] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937541.990002] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937541.996622] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.002896] ksoftirqd/29 S ffff8b5da98fa080 0 158 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.010165] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.012795] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.017945] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.023958] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.029972] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.035033] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.041309] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.047931] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.054204] kworker/29:0H S ffff8b5da98fc100 0 160 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.061489] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.064122] [] ? start_worker+0x27/0x30 [Fri Apr 12 11:10:20 2019][2937542.069788] [] ? manage_workers.isra.25+0x189/0x2a0 [Fri Apr 12 11:10:20 2019][2937542.076497] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.081644] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:20 2019][2937542.087400] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:20 2019][2937542.094107] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.099169] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.105443] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.112064] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.118338] watchdog/30 S ffff8b6da9976180 0 161 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.125607] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.128240] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:20 2019][2937542.134427] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:20 2019][2937542.139922] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.145067] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.151075] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.157088] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.162142] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.168419] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.175038] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.181312] migration/30 S ffff8b6da9975140 0 162 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.188581] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.191212] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:20 2019][2937542.197834] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.202981] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.208997] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.215011] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.220073] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.226347] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.232968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.239242] ksoftirqd/30 S ffff8b6da99a0000 0 163 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.246519] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.249152] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.254297] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.260312] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.266326] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.271386] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.277656] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.284275] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.290540] kworker/30:0H S ffff8b6da99a2080 0 165 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.297814] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.300449] [] ? start_worker+0x27/0x30 [Fri Apr 12 11:10:20 2019][2937542.306118] [] ? manage_workers.isra.25+0x189/0x2a0 [Fri Apr 12 11:10:20 2019][2937542.312825] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.317972] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:20 2019][2937542.323727] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:20 2019][2937542.330434] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.335497] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.341773] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.348394] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.354668] watchdog/31 S ffff8b7da9606180 0 166 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.361943] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.364574] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:20 2019][2937542.370070] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.375219] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.381231] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.387245] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.392299] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.398576] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.405198] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.411484] migration/31 S ffff8b7da9605140 0 167 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.418756] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.421385] [] ? cpu_stop_unpark+0x50/0x50 [Fri Apr 12 11:10:20 2019][2937542.427315] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.432462] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.438476] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.444491] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.449553] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.455828] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.462450] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.468722] ksoftirqd/31 S ffff8b7da9630000 0 168 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.475992] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.478622] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.483771] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.489783] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.495790] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.500852] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.507126] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.513747] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.520023] kworker/31:0H S ffff8b7da9632080 0 170 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.527305] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.529939] [] ? start_worker+0x27/0x30 [Fri Apr 12 11:10:20 2019][2937542.535605] [] ? manage_workers.isra.25+0x189/0x2a0 [Fri Apr 12 11:10:20 2019][2937542.542308] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.547454] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:20 2019][2937542.553210] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:20 2019][2937542.559916] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.564979] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.571252] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.577876] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.584163] watchdog/32 S ffff8b4e29659040 0 171 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.591433] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.594066] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:20 2019][2937542.599559] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.604707] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.610721] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.616737] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.621797] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.628072] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.634695] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.640983] migration/32 S ffff8b4e296d8000 0 172 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.648254] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.650885] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.656032] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.662048] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.668062] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.673123] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.679398] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.686020] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.692295] ksoftirqd/32 S ffff8b4e296d9040 0 173 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.699571] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.702203] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.707348] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.713354] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.719370] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.724432] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.730705] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.737329] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.743600] kworker/32:0H S ffff8b4e296db0c0 0 175 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.750876] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.753511] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:20 2019][2937542.759695] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.764836] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:20 2019][2937542.770591] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:20 2019][2937542.777297] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.782351] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.788628] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.795246] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.801514] watchdog/33 S ffff8b5da98fe180 0 176 2 0x00000000 [Fri Apr 12 11:10:20 2019][2937542.808791] Call Trace: [Fri Apr 12 11:10:20 2019][2937542.811424] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:20 2019][2937542.816918] [] schedule+0x29/0x70 [Fri Apr 12 11:10:20 2019][2937542.822065] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:20 2019][2937542.828080] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.834093] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:20 2019][2937542.839155] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:20 2019][2937542.845429] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:20 2019][2937542.852051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937542.858324] migration/33 S ffff8b5da98fd140 0 177 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937542.865603] Call Trace: [Fri Apr 12 11:10:21 2019][2937542.868235] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937542.873394] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937542.879415] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937542.885447] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937542.890516] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937542.896790] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937542.903409] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937542.909675] ksoftirqd/33 S ffff8b5da9930000 0 178 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937542.916945] Call Trace: [Fri Apr 12 11:10:21 2019][2937542.919580] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937542.924745] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937542.930758] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937542.936769] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937542.941834] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937542.948117] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937542.954736] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937542.961010] kworker/33:0H S ffff8b5da9932080 0 180 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937542.968302] Call Trace: [Fri Apr 12 11:10:21 2019][2937542.970939] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:21 2019][2937542.977124] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937542.982276] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:21 2019][2937542.988035] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:21 2019][2937542.994762] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937542.999830] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.006103] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.012725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.018998] watchdog/34 S ffff8b6da99a6180 0 181 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.026270] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.028906] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:21 2019][2937543.034401] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.039549] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.045563] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.051570] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.056633] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.062910] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.069529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.075802] migration/34 S ffff8b6da99a5140 0 182 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.083070] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.085702] [] ? cpu_stop_queue_work+0x70/0x70 [Fri Apr 12 11:10:21 2019][2937543.091976] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.097141] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.103154] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.109162] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.114223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.120499] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.127118] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.133385] ksoftirqd/34 S ffff8b6da99a4100 0 183 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.140662] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.143292] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.148440] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.154454] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.160461] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.165520] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.171790] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.178409] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.184675] kworker/34:0H S ffff8b6da99e0000 0 185 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.191952] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.194585] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:21 2019][2937543.200771] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.205919] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:21 2019][2937543.211687] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:21 2019][2937543.218391] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.223452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.229728] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.236348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.242622] watchdog/35 S ffff8b7da9636180 0 186 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.249890] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.252521] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:21 2019][2937543.258709] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:21 2019][2937543.264203] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.269370] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.275385] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.281398] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.286459] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.292736] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.299357] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.305631] migration/35 S ffff8b7da9635140 0 187 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.312910] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.315538] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.320687] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.326699] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.332704] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.337760] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.344037] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.350655] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.356928] ksoftirqd/35 S ffff8b7da9634100 0 188 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.364197] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.366830] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.371976] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.377991] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.384006] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.389066] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.395334] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.401955] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.408228] kworker/35:0H S ffff8b7da9668000 0 190 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.415503] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.418139] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:21 2019][2937543.424325] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.429474] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:21 2019][2937543.435228] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:21 2019][2937543.441934] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.446995] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.453264] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.459884] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.466159] watchdog/36 S ffff8b4e296de180 0 191 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.473435] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.476066] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:21 2019][2937543.482252] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:21 2019][2937543.487747] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.492897] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.498915] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.504925] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.509987] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.516261] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.522883] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.529157] migration/36 S ffff8b4e296dd140 0 192 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.536425] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.539057] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:21 2019][2937543.545677] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.550828] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.556853] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.562863] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.567924] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.574201] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.580821] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.587096] ksoftirqd/36 S ffff8b4e296dc100 0 193 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.594372] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.597003] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.602152] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.608164] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.614194] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.619250] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.625527] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.632146] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.638420] kworker/36:0H S ffff8b4e29731040 0 195 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.645696] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.648329] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:21 2019][2937543.654515] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.659665] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:21 2019][2937543.665418] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:21 2019][2937543.672128] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.677187] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.683465] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.690084] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.696357] watchdog/37 S ffff8b5da9936180 0 196 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.703628] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.706259] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:21 2019][2937543.711759] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.716918] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.722930] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.728938] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.733998] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.740265] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.746887] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.753159] migration/37 S ffff8b5da9935140 0 197 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.760428] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.763061] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:21 2019][2937543.769682] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.774828] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.780844] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.786858] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.791921] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.798193] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.804808] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.811083] ksoftirqd/37 S ffff8b5da9934100 0 198 2 0x00000000 [Fri Apr 12 11:10:21 2019][2937543.818360] Call Trace: [Fri Apr 12 11:10:21 2019][2937543.820990] [] schedule+0x29/0x70 [Fri Apr 12 11:10:21 2019][2937543.826137] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:21 2019][2937543.832152] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.838167] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:21 2019][2937543.843227] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:21 2019][2937543.849504] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:21 2019][2937543.856124] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937543.862399] kworker/37:0H S ffff8b5da9960000 0 200 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937543.869681] Call Trace: [Fri Apr 12 11:10:22 2019][2937543.872316] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:22 2019][2937543.878504] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937543.883651] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:22 2019][2937543.889406] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:22 2019][2937543.896115] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937543.901176] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937543.907451] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937543.914071] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937543.920346] watchdog/38 S ffff8b6da99e6180 0 201 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937543.927622] Call Trace: [Fri Apr 12 11:10:22 2019][2937543.930253] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:22 2019][2937543.935747] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937543.940895] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937543.946909] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937543.952924] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937543.957984] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937543.964252] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937543.970873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937543.977148] migration/38 S ffff8b6da99e5140 0 202 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937543.984415] Call Trace: [Fri Apr 12 11:10:22 2019][2937543.987046] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:22 2019][2937543.993668] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937543.998813] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.004822] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.010837] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.015913] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.022183] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.028804] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.035076] ksoftirqd/38 S ffff8b6da99e4100 0 203 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.042373] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.045006] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.050151] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.056186] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.062197] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.067259] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.073535] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.080156] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.086427] kworker/38:0H S ffff8b6da99e2080 0 205 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.093701] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.096336] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:22 2019][2937544.102524] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.107675] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:22 2019][2937544.113430] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:22 2019][2937544.120137] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.125214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.131507] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.138129] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.144400] watchdog/39 S ffff8b7da966e180 0 206 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.151670] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.154303] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:22 2019][2937544.159797] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.164943] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.170957] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.176972] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.182033] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.188308] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.194932] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.201204] migration/39 S ffff8b7da966d140 0 207 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.208480] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.211112] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:22 2019][2937544.217731] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.222881] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.228894] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.234903] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.239984] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.246260] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.252881] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.259153] ksoftirqd/39 S ffff8b7da966c100 0 208 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.266454] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.269085] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.274234] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.280250] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.286271] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.291332] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.297608] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.304228] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.310502] kworker/39:0H S ffff8b7da966a080 0 210 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.317802] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.320437] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:22 2019][2937544.326626] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.331775] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:22 2019][2937544.337528] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:22 2019][2937544.344236] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.349293] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.355574] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.362195] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.368468] watchdog/40 S ffff8b4e29736180 0 211 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.375786] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.378418] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:22 2019][2937544.384628] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:22 2019][2937544.390117] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.395266] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.401281] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.407298] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.412363] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.418651] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.425271] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.431570] migration/40 S ffff8b4e29735140 0 212 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.438846] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.441478] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.446624] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.452640] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.458655] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.463715] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.469996] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.476615] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.482904] ksoftirqd/40 S ffff8b4e29734100 0 213 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.490200] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.492860] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.498007] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.504036] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.510048] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.515111] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.521387] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.528015] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.534334] kworker/40:0H S ffff8b4e29732080 0 215 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.541677] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.544315] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:22 2019][2937544.550930] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.556485] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:22 2019][2937544.562245] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:22 2019][2937544.568961] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.574034] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.580323] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.586979] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.593260] watchdog/41 S ffff8b5da9966180 0 216 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.600561] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.603202] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:22 2019][2937544.608698] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.613844] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.619858] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.625877] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.630941] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.637221] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.643852] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.650124] migration/41 S ffff8b5da9965140 0 217 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.657410] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.660062] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:22 2019][2937544.666688] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.671837] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.677849] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.683863] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.688926] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.695200] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.701822] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.708095] ksoftirqd/41 S ffff8b5da9964100 0 218 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.715381] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.718013] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.723160] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.729173] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.735191] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.740251] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.746528] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.753147] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.759421] kworker/41:0H S ffff8b5da9962080 0 220 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.766712] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.769348] [] ? start_worker+0x27/0x30 [Fri Apr 12 11:10:22 2019][2937544.775015] [] ? manage_workers.isra.25+0x189/0x2a0 [Fri Apr 12 11:10:22 2019][2937544.781721] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.786870] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:22 2019][2937544.792625] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:22 2019][2937544.799333] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.804393] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.810669] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:22 2019][2937544.817289] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.823563] watchdog/42 S ffff8b6da99e1040 0 221 2 0x00000000 [Fri Apr 12 11:10:22 2019][2937544.830839] Call Trace: [Fri Apr 12 11:10:22 2019][2937544.833473] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:22 2019][2937544.838967] [] schedule+0x29/0x70 [Fri Apr 12 11:10:22 2019][2937544.844113] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:22 2019][2937544.850130] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:22 2019][2937544.856143] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:22 2019][2937544.861204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937544.867481] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937544.874102] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937544.880375] migration/42 S ffff8b6da9a40000 0 222 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937544.887653] Call Trace: [Fri Apr 12 11:10:23 2019][2937544.890283] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937544.895432] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937544.901445] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937544.907460] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937544.912520] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937544.918797] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937544.925418] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937544.931692] ksoftirqd/42 S ffff8b6da9a41040 0 223 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937544.938994] Call Trace: [Fri Apr 12 11:10:23 2019][2937544.941626] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937544.946775] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937544.952787] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937544.958803] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937544.963864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937544.970140] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937544.976760] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937544.983034] kworker/42:0H S ffff8b6da9a430c0 0 225 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937544.990316] Call Trace: [Fri Apr 12 11:10:23 2019][2937544.992952] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:23 2019][2937544.999141] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.004286] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:23 2019][2937545.010044] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:23 2019][2937545.016749] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.021811] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.028085] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.034709] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.040981] watchdog/43 S ffff8b7da9669040 0 226 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.048256] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.050888] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:23 2019][2937545.057078] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:23 2019][2937545.062571] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.067721] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.073733] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.079749] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.084810] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.091087] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.097706] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.103980] migration/43 S ffff8b7da96c8000 0 227 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.111281] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.113914] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.119064] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.125074] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.131096] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.136152] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.142429] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.149048] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.155323] ksoftirqd/43 S ffff8b7da96c9040 0 228 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.162599] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.165230] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.170380] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.176392] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.182407] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.187466] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.193744] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.200364] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.206640] kworker/43:0H S ffff8b7da96cb0c0 0 230 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.213921] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.216556] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:23 2019][2937545.222742] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.227893] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:23 2019][2937545.233647] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:23 2019][2937545.240355] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.245415] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.251690] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.258313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.264585] watchdog/44 S ffff8b4e297a0000 0 231 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.271863] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.274497] [] ? native_sched_clock_from_tsc+0x70/0x70 [Fri Apr 12 11:10:23 2019][2937545.281462] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:23 2019][2937545.286956] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.292107] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.298119] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.304136] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.309194] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.315477] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.322114] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.328391] migration/44 S ffff8b4e297a1040 0 232 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.335676] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.338309] [] ? cpu_stop_unpark+0x50/0x50 [Fri Apr 12 11:10:23 2019][2937545.344238] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.349387] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.355397] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.361413] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.366484] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.372759] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.379381] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.385655] ksoftirqd/44 S ffff8b4e297a2080 0 233 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.392930] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.395564] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.400710] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.406725] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.412739] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.417800] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.424073] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.430696] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.436969] kworker/44:0H S ffff8b4e297a4100 0 235 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.444270] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.446909] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:23 2019][2937545.453101] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.458249] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:23 2019][2937545.464004] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:23 2019][2937545.470711] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.475772] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.482047] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.488672] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.494943] watchdog/45 S ffff8b5da9961040 0 236 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.502271] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.504905] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:23 2019][2937545.510398] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.515547] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.521557] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.527571] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.532627] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.538902] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.545526] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.551798] migration/45 S ffff8b5da99c8000 0 237 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.559073] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.561706] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.566851] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.572882] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.578897] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.583960] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.590234] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.596856] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.603129] ksoftirqd/45 S ffff8b5da99c9040 0 238 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.610407] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.613047] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.618193] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.624209] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.630255] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.635322] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.641593] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.648215] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.654488] kworker/45:0H S ffff8b5da99cb0c0 0 240 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.661764] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.664399] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:23 2019][2937545.670587] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.675736] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:23 2019][2937545.681489] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:23 2019][2937545.688194] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.693256] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.699546] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.706172] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.712463] watchdog/46 S ffff8b6da9a46180 0 241 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.719739] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.722372] [] ? touch_nmi_watchdog+0x30/0x30 [Fri Apr 12 11:10:23 2019][2937545.728558] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:23 2019][2937545.734053] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.739200] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.745215] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.751230] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.756293] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.762569] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.769190] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.775489] migration/46 S ffff8b6da9a45140 0 242 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.782785] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.785425] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.790572] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.796587] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.802602] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.807681] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.813956] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:23 2019][2937545.820582] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.826856] ksoftirqd/46 S ffff8b6da9a44100 0 243 2 0x00000000 [Fri Apr 12 11:10:23 2019][2937545.834137] Call Trace: [Fri Apr 12 11:10:23 2019][2937545.836783] [] schedule+0x29/0x70 [Fri Apr 12 11:10:23 2019][2937545.841934] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:23 2019][2937545.847963] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:23 2019][2937545.853985] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:23 2019][2937545.859048] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937545.865325] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937545.871950] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937545.878231] kworker/46:0H S ffff8b6da9a79040 0 245 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937545.885508] Call Trace: [Fri Apr 12 11:10:24 2019][2937545.888155] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:24 2019][2937545.894341] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937545.899990] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:24 2019][2937545.906238] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:24 2019][2937545.913017] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937545.918076] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937545.924363] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937545.930982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937545.937254] watchdog/47 S ffff8b7da96ce180 0 246 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937545.944531] Call Trace: [Fri Apr 12 11:10:24 2019][2937545.947163] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:24 2019][2937545.952656] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937545.957807] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:24 2019][2937545.963821] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937545.969836] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937545.974906] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937545.981191] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937545.987812] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937545.994084] migration/47 S ffff8b7da96cd140 0 247 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.001370] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.004004] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.009152] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:24 2019][2937546.015163] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.021188] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.026258] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.032543] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.039170] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.045444] ksoftirqd/47 S ffff8b7da96cc100 0 248 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.052720] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.055354] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.060499] [] smpboot_thread_fn+0xe2/0x1a0 [Fri Apr 12 11:10:24 2019][2937546.066512] [] ? lg_double_unlock+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.072531] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.077588] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.083856] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.090498] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.096769] kworker/47:0H S ffff8b7da9701040 0 250 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.104051] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.106686] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:24 2019][2937546.112873] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.118018] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.123766] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:24 2019][2937546.130475] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.135547] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.141848] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.148479] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.154752] kdevtmpfs S ffff8b4e2909a080 0 255 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.162036] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.164669] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.169815] [] devtmpfsd+0x16d/0x180 [Fri Apr 12 11:10:24 2019][2937546.175221] [] ? handle_create.isra.2+0x220/0x220 [Fri Apr 12 11:10:24 2019][2937546.181758] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.186816] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.193083] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.199709] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.205989] netns S ffff8b4e2909b0c0 0 256 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.213272] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.215906] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.221051] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.226899] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.233082] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.238154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.244436] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.251056] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.257329] khungtaskd S ffff8b4e2909c100 0 257 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.264600] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.267229] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.272375] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:24 2019][2937546.278382] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:10:24 2019][2937546.284045] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:24 2019][2937546.290581] [] schedule_timeout_interruptible+0x1e/0x20 [Fri Apr 12 11:10:24 2019][2937546.297645] [] watchdog+0x4c/0x2c0 [Fri Apr 12 11:10:24 2019][2937546.302878] [] ? reset_hung_task_detector+0x20/0x20 [Fri Apr 12 11:10:24 2019][2937546.309614] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.314691] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.320977] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.327615] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.333885] writeback S ffff8b4e2909d140 0 258 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.341169] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.343803] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.348953] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.354793] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.360980] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.366040] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.372322] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.378957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.385229] kintegrityd S ffff8b4e2909e180 0 259 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.392515] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.395174] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.400320] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.406164] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.412358] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.417421] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.423704] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.430323] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.436596] bioset S ffff8b4e29099040 0 260 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.443925] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.446559] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.451710] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.457569] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.463762] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.468823] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.475097] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.481718] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.487991] bioset S ffff8b4e29098000 0 261 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.495284] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.497927] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.503075] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.508913] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.515102] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.520164] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.526439] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.533059] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.539332] bioset S ffff8b5cfefb8000 0 262 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.546607] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.549242] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.554390] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.560229] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.566418] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.571478] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.577748] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.584366] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.590641] kblockd S ffff8b5cfefb9040 0 263 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.597922] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.600560] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.605707] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.611547] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.617758] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.622846] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.629146] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.635766] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.642036] md S ffff8b5cfefba080 0 264 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.649330] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.651963] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.657119] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.662963] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.669148] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.674210] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.680489] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.687115] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.693387] edac-poller S ffff8b5cfefbb0c0 0 265 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.700678] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.703312] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.708503] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.714376] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.720572] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.725632] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.731909] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.738526] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.744799] watchdogd S ffff8b5cfefbc100 0 266 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.752081] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.754717] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.759865] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:24 2019][2937546.765709] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:24 2019][2937546.771894] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.776987] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.783308] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.790257] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.796908] irq/26-AMD-Vi S ffff8b5da99cc100 0 269 2 0x00000000 [Fri Apr 12 11:10:24 2019][2937546.804192] Call Trace: [Fri Apr 12 11:10:24 2019][2937546.806835] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Fri Apr 12 11:10:24 2019][2937546.813892] [] schedule+0x29/0x70 [Fri Apr 12 11:10:24 2019][2937546.819054] [] irq_thread+0xd1/0x180 [Fri Apr 12 11:10:24 2019][2937546.824472] [] ? irq_forced_thread_fn+0x50/0x50 [Fri Apr 12 11:10:24 2019][2937546.830838] [] ? irq_thread_check_affinity+0xf0/0xf0 [Fri Apr 12 11:10:24 2019][2937546.837643] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:24 2019][2937546.842779] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.849060] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:24 2019][2937546.855697] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:24 2019][2937546.861966] irq/27-AMD-Vi S ffff8b5cfefbe180 0 270 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937546.869261] Call Trace: [Fri Apr 12 11:10:25 2019][2937546.871894] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Fri Apr 12 11:10:25 2019][2937546.878950] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937546.884117] [] irq_thread+0xd1/0x180 [Fri Apr 12 11:10:25 2019][2937546.889531] [] ? irq_forced_thread_fn+0x50/0x50 [Fri Apr 12 11:10:25 2019][2937546.895902] [] ? irq_thread_check_affinity+0xf0/0xf0 [Fri Apr 12 11:10:25 2019][2937546.902708] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937546.907806] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937546.914103] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937546.920733] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937546.927004] irq/28-AMD-Vi S ffff8b5cfefbd140 0 271 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937546.934296] Call Trace: [Fri Apr 12 11:10:25 2019][2937546.936943] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Fri Apr 12 11:10:25 2019][2937546.944003] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937546.949169] [] irq_thread+0xd1/0x180 [Fri Apr 12 11:10:25 2019][2937546.954583] [] ? irq_forced_thread_fn+0x50/0x50 [Fri Apr 12 11:10:25 2019][2937546.960955] [] ? irq_thread_check_affinity+0xf0/0xf0 [Fri Apr 12 11:10:25 2019][2937546.967768] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937546.972826] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937546.979122] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937546.985739] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937546.992013] irq/29-AMD-Vi S ffff8b5cfc120000 0 272 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937546.999299] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.001930] [] ? irq_finalize_oneshot.part.31+0xf0/0xf0 [Fri Apr 12 11:10:25 2019][2937547.008987] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.014162] [] irq_thread+0xd1/0x180 [Fri Apr 12 11:10:25 2019][2937547.019565] [] ? irq_forced_thread_fn+0x50/0x50 [Fri Apr 12 11:10:25 2019][2937547.025925] [] ? irq_thread_check_affinity+0xf0/0xf0 [Fri Apr 12 11:10:25 2019][2937547.032725] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.037782] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.044061] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.050679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.056993] kswapd0 S ffff8b5cfc125140 0 277 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.064265] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.066898] [] ? calculate_normal_threshold+0x27/0x50 [Fri Apr 12 11:10:25 2019][2937547.073777] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.078929] [] kswapd+0x38f/0x440 [Fri Apr 12 11:10:25 2019][2937547.084076] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:25 2019][2937547.090086] [] ? balance_pgdat+0x5e0/0x5e0 [Fri Apr 12 11:10:25 2019][2937547.096017] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.101076] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.107356] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.113976] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.120247] kswapd1 S ffff8b5cfc126180 0 278 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.127533] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.130166] [] ? calculate_normal_threshold+0x27/0x50 [Fri Apr 12 11:10:25 2019][2937547.137046] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.142194] [] kswapd+0x38f/0x440 [Fri Apr 12 11:10:25 2019][2937547.147342] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:25 2019][2937547.153356] [] ? balance_pgdat+0x5e0/0x5e0 [Fri Apr 12 11:10:25 2019][2937547.159282] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.164344] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.170619] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.177241] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.183517] kswapd2 S ffff8b5cfc200000 0 279 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.190801] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.193434] [] ? calculate_normal_threshold+0x27/0x50 [Fri Apr 12 11:10:25 2019][2937547.200315] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.205463] [] kswapd+0x38f/0x440 [Fri Apr 12 11:10:25 2019][2937547.210617] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:25 2019][2937547.216640] [] ? balance_pgdat+0x5e0/0x5e0 [Fri Apr 12 11:10:25 2019][2937547.222572] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.227631] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.233906] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.240529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.246800] kswapd3 S ffff8b5cfc201040 0 280 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.254069] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.256701] [] ? calculate_normal_threshold+0x27/0x50 [Fri Apr 12 11:10:25 2019][2937547.263590] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.268750] [] kswapd+0x38f/0x440 [Fri Apr 12 11:10:25 2019][2937547.273895] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:25 2019][2937547.279919] [] ? balance_pgdat+0x5e0/0x5e0 [Fri Apr 12 11:10:25 2019][2937547.285848] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.290911] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.297200] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.303823] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.310095] ksmd S ffff8b5cfc202080 0 281 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.317371] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.320003] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.325169] [] ksm_scan_thread+0x16a/0x240 [Fri Apr 12 11:10:25 2019][2937547.331096] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:25 2019][2937547.337112] [] ? ksm_do_scan+0x11b0/0x11b0 [Fri Apr 12 11:10:25 2019][2937547.343058] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.348118] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.354391] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.361023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.367295] khugepaged S ffff8b5cfc2030c0 0 282 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.374575] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.377207] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.382352] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:25 2019][2937547.388366] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:25 2019][2937547.394905] [] khugepaged+0x226/0x480 [Fri Apr 12 11:10:25 2019][2937547.400397] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:25 2019][2937547.406413] [] ? khugepaged_scan_mm_slot+0xf00/0xf00 [Fri Apr 12 11:10:25 2019][2937547.413215] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.418275] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.424551] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.431171] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.437444] crypto S ffff8b5cfc204100 0 283 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.444721] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.447354] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.452502] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:25 2019][2937547.458344] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:25 2019][2937547.464531] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.469591] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.475867] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.482489] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.488772] kthrotld S ffff8b5cfc25d140 0 291 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.496055] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.498689] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.503839] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:25 2019][2937547.509694] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:25 2019][2937547.515884] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.520943] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.527252] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.533875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.540149] kmpath_rdacd S ffff8b6cfcdd1040 0 296 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.547434] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.550068] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.555222] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:25 2019][2937547.561063] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:25 2019][2937547.567249] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.572313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.578588] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.585218] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.591491] kaluad S ffff8b6cfcdd2080 0 297 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.598774] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.601408] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.606556] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:25 2019][2937547.612400] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:25 2019][2937547.618584] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.623647] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.629923] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.636544] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.642819] kpsmoused S ffff8b5cfc2b6180 0 301 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.650117] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.652751] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.657907] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:25 2019][2937547.663769] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:25 2019][2937547.669953] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.675014] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.681289] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.687908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.694177] ipv6_addrconf S ffff8b5cfc2b4100 0 303 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.701475] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.704111] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.709257] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:25 2019][2937547.715099] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:25 2019][2937547.721289] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.726350] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.732624] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.739246] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.745518] deferwq S ffff8b5cfbc79040 0 317 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.752817] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.755455] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.760612] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:25 2019][2937547.766452] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:25 2019][2937547.772639] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.777720] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.783996] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.790611] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.796896] kauditd S ffff8b6cfc4b9040 0 421 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.804172] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.806814] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.811960] [] kauditd_thread+0x184/0x1e0 [Fri Apr 12 11:10:25 2019][2937547.817801] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:25 2019][2937547.823554] [] ? audit_printk_skb+0x70/0x70 [Fri Apr 12 11:10:25 2019][2937547.829562] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:25 2019][2937547.834622] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.840898] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:25 2019][2937547.847519] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:25 2019][2937547.853793] scsi_eh_0 S ffff8b7cf9b49040 0 1956 2 0x00000000 [Fri Apr 12 11:10:25 2019][2937547.861060] Call Trace: [Fri Apr 12 11:10:25 2019][2937547.863691] [] schedule+0x29/0x70 [Fri Apr 12 11:10:25 2019][2937547.868841] [] scsi_error_handler+0xa3/0x8b0 [Fri Apr 12 11:10:26 2019][2937547.874941] [] ? scsi_eh_get_sense+0x250/0x250 [Fri Apr 12 11:10:26 2019][2937547.881215] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937547.886276] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937547.892554] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937547.899174] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937547.905447] scsi_tmf_0 S ffff8b7cf9b4e180 0 1957 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937547.912729] Call Trace: [Fri Apr 12 11:10:26 2019][2937547.915363] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937547.920513] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:26 2019][2937547.926372] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:26 2019][2937547.932559] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937547.937619] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937547.943904] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937547.950531] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937547.956800] ata_sff S ffff8b5cf6cbc100 0 3863 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937547.964090] Call Trace: [Fri Apr 12 11:10:26 2019][2937547.966724] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937547.971874] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:26 2019][2937547.977714] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:26 2019][2937547.983904] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937547.988982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937547.995260] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.001888] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.008161] poll_mpt3sas0_h S ffff8b5cf6ba5140 0 5276 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.015442] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.018077] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.023226] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.029066] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:26 2019][2937548.035269] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.040358] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.046635] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.053254] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.059536] scsi_eh_1 S ffff8b5cf660b0c0 0 5579 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.066823] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.069464] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.074610] [] scsi_error_handler+0xa3/0x8b0 [Fri Apr 12 11:10:26 2019][2937548.080712] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:10:26 2019][2937548.087688] [] ? scsi_eh_get_sense+0x250/0x250 [Fri Apr 12 11:10:26 2019][2937548.093963] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.099023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.105311] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.111927] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.118201] scsi_tmf_1 S ffff8b5cf660c100 0 5590 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.125476] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.128117] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.133268] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.139123] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:26 2019][2937548.145310] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.150365] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.156640] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.163262] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.169534] scsi_eh_2 S ffff8b7cf32b1040 0 7254 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.176805] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.179437] [] ? __scsi_iterate_devices+0x68/0x90 [Fri Apr 12 11:10:26 2019][2937548.185979] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.191144] [] scsi_error_handler+0xa3/0x8b0 [Fri Apr 12 11:10:26 2019][2937548.197246] [] ? scsi_eh_get_sense+0x250/0x250 [Fri Apr 12 11:10:26 2019][2937548.203540] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.208608] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.214883] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.221503] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.227780] scsi_tmf_2 S ffff8b7cf3006180 0 7255 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.235080] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.237724] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.242871] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.248718] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:26 2019][2937548.254927] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.259986] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.266262] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.272882] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.279155] mlx5_health0000 S ffff8b7cf3252080 0 7283 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.286439] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.289070] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.294220] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.300062] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:26 2019][2937548.306249] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.311310] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.317586] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.324206] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.330496] mlx5_cmd_0000:8 S ffff8b7cf3250000 0 7285 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.337813] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.340449] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.345598] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.351439] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:26 2019][2937548.357624] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.362726] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.368997] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.375618] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.381890] kworker/14:1 S ffff8b7cf3301040 0 7296 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.389166] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.391800] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:26 2019][2937548.398012] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.403174] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.408935] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:26 2019][2937548.415664] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.420720] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.426996] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.433616] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.439891] kworker/26:1 S ffff8b7cf3306180 0 7304 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.447219] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.449861] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:26 2019][2937548.456053] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.461204] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.466959] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:26 2019][2937548.473675] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.478734] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.485005] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.491624] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.497934] kworker/31:1 S ffff8b8cfc81b0c0 0 7306 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.505243] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.507876] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:26 2019][2937548.514082] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.519229] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.524985] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:26 2019][2937548.531701] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.536759] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.543038] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.549658] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.555940] kworker/47:1 S ffff8b8cfc81a080 0 7314 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.563222] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.565859] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:26 2019][2937548.572043] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.577190] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.582946] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:26 2019][2937548.589654] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.594715] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.600991] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.607612] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.613885] mlx5_page_alloc S ffff8b7cf99b5140 0 7348 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.621183] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.623821] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.628971] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.634811] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:26 2019][2937548.641022] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.646096] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.652389] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.659011] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.665288] kworker/u261:0 S ffff8b7cf99b2080 0 7394 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.673339] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.676001] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.681200] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.686959] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:26 2019][2937548.693664] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.698727] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.705032] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.711673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.717949] kworker/u262:0 S ffff8b5cfc258000 0 7395 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.725242] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.727875] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.733031] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.738787] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:26 2019][2937548.745502] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.750562] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.756843] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.763908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.770270] kworker/u263:0 S ffff8b6cf75eb0c0 0 7396 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.777574] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.780208] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.785390] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.791176] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:26 2019][2937548.797895] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.802980] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.809259] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.815881] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.822170] kworker/u265:0 S ffff8b8cf4736180 0 7398 2 0x00000000 [Fri Apr 12 11:10:26 2019][2937548.829489] Call Trace: [Fri Apr 12 11:10:26 2019][2937548.832122] [] schedule+0x29/0x70 [Fri Apr 12 11:10:26 2019][2937548.837271] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:26 2019][2937548.843036] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:26 2019][2937548.849738] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:26 2019][2937548.854801] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:26 2019][2937548.861076] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:26 2019][2937548.867721] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937548.873996] mlx5_fc S ffff8b7cf3036180 0 7399 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937548.881281] Call Trace: [Fri Apr 12 11:10:27 2019][2937548.883914] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937548.889061] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937548.894900] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937548.901091] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937548.906154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937548.912438] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937548.919068] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937548.925342] ib-comp-wq S ffff8b7cf30030c0 0 7412 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937548.932627] Call Trace: [Fri Apr 12 11:10:27 2019][2937548.935274] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937548.940421] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937548.946279] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937548.952487] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937548.957551] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937548.963826] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937548.970452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937548.976998] ib-comp-unb-wq S ffff8b7cf3000000 0 7413 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937548.984306] Call Trace: [Fri Apr 12 11:10:27 2019][2937548.986942] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937548.992114] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937548.997976] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.004212] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.009352] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.015626] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.022285] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.028572] ib_mcast S ffff8b7cf3001040 0 7414 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.036368] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.038999] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.044299] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.050142] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.056330] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.061394] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.067668] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.074361] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.080635] ib_nl_sa_wq S ffff8b7cf3002080 0 7415 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.087914] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.090550] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.095717] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.101558] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.107743] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.112803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.119078] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.125698] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.131974] mlx5_ib_sigerr_ S ffff8b7cf99b6180 0 7423 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.139258] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.141939] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.147093] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.152953] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.159158] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.164215] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.170506] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.177145] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.183459] ib_mad1 S ffff8b7cf9b4b0c0 0 7428 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.190768] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.193414] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.198569] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.204436] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.210633] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.215697] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.221979] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.228604] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.234874] to_fifo S ffff8b7cf9b4d140 0 7429 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.242149] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.244782] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.249940] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.255782] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.261968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.267032] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.273305] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.279924] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.286195] mkey_cache S ffff8b7cf99b4100 0 7432 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.293476] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.296108] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.301256] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.307112] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.313292] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.318350] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.324625] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.331262] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.337534] kworker/15:1H S ffff8b7da9704100 0 7448 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.344833] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.347473] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:27 2019][2937549.353654] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.358805] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.364559] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:27 2019][2937549.371267] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.376371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.382651] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.389289] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.395580] kworker/0:1H S ffff8b5cfc25c100 0 7450 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.402884] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.405541] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:27 2019][2937549.411782] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.416948] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.422720] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:27 2019][2937549.429452] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.434511] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.440873] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.447569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.453862] poll_mpt3sas0_s S ffff8b5cfc25b0c0 0 7451 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.461155] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.463808] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.468954] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.474795] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.480992] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.486051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.492327] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.498958] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.505239] jbd2/sdk2-8 S ffff8b7cf9892080 0 7793 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.512515] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.515149] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.520331] [] kjournald2+0x203/0x260 [jbd2] [Fri Apr 12 11:10:27 2019][2937549.526432] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:27 2019][2937549.532456] [] ? commit_timeout+0x10/0x10 [jbd2] [Fri Apr 12 11:10:27 2019][2937549.538902] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.543983] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.550276] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.557350] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.563878] ext4-rsv-conver S ffff8b7cf9890000 0 7794 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.571163] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.573836] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.579007] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:27 2019][2937549.584864] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:27 2019][2937549.591084] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.596148] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.602423] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.609453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.615858] systemd-journal R running task 0 7867 1 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.623140] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.625794] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:10:27 2019][2937549.631466] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.636622] [] ? schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:27 2019][2937549.644147] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Fri Apr 12 11:10:27 2019][2937549.651157] [] ? ep_poll+0x123/0x360 [Fri Apr 12 11:10:27 2019][2937549.656593] [] ? do_truncate+0x88/0xc0 [Fri Apr 12 11:10:27 2019][2937549.662171] [] ? __sb_start_write+0x58/0x110 [Fri Apr 12 11:10:27 2019][2937549.668271] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:27 2019][2937549.674057] [] ? SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:27 2019][2937549.679991] [] ? system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:27 2019][2937549.686362] systemd-udevd S ffff8b7cf328d140 0 7899 1 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.693637] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.696268] [] ? kmem_cache_alloc+0x35/0x1f0 [Fri Apr 12 11:10:27 2019][2937549.702386] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.707549] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:27 2019][2937549.714790] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Fri Apr 12 11:10:27 2019][2937549.721755] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:27 2019][2937549.728290] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:27 2019][2937549.733527] [] ? do_unlinkat+0xf1/0x2d0 [Fri Apr 12 11:10:27 2019][2937549.739223] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:27 2019][2937549.744984] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:27 2019][2937549.750737] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:27 2019][2937549.756931] ccp-1-q2 S ffff8b5cf32dd140 0 12388 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.764210] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.766842] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.772548] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:27 2019][2937549.779401] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:27 2019][2937549.786146] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:27 2019][2937549.792871] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.797933] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.804372] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:27 2019][2937549.811056] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:27 2019][2937549.817341] ccp-1-q3 S ffff8b5cf32de180 0 12389 2 0x00000000 [Fri Apr 12 11:10:27 2019][2937549.824634] Call Trace: [Fri Apr 12 11:10:27 2019][2937549.827267] [] schedule+0x29/0x70 [Fri Apr 12 11:10:27 2019][2937549.832451] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:27 2019][2937549.839249] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:27 2019][2937549.845959] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:27 2019][2937549.852668] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:27 2019][2937549.857751] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937549.864050] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937549.870680] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937549.876952] ccp-1-q4 S ffff8b5cf93d8000 0 12395 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937549.884231] Call Trace: [Fri Apr 12 11:10:28 2019][2937549.886879] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:28 2019][2937549.892392] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937549.897567] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937549.904385] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937549.911809] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937549.918613] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937549.923678] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937549.930050] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937549.936759] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937549.943041] ccp-2-q0 S ffff8b5cf5d92080 0 13496 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937549.950333] Call Trace: [Fri Apr 12 11:10:28 2019][2937549.952984] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:28 2019][2937549.958493] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937549.963656] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937549.970504] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937549.977225] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937549.983929] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937549.988990] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937549.995266] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.001899] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.008562] ccp-2-q1 S ffff8b5cfab50000 0 13499 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.015888] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.018536] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.023698] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.030490] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.037218] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.043942] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.049013] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.055295] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.061922] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.068203] ccp-2-q2 S ffff8b5cfab51040 0 13500 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.075481] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.078162] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.083343] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.090154] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.096890] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.103626] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.108688] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.114964] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.121612] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.127903] ccp-2-q3 S ffff8b5cfab52080 0 13502 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.135205] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.137837] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:28 2019][2937550.143352] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.148523] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.155340] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.162099] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.168825] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.173890] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.180167] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.186784] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.193076] ccp-2-q4 S ffff8b5cfb302080 0 13504 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.200353] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.202985] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.208147] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.214976] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.221727] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.228432] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.233511] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.239784] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.246415] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.252685] ccp-3-q2 S ffff8b5cfb306180 0 13519 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.259974] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.262616] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.267776] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.274572] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.281279] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.287987] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.293051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.299323] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.305942] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.312227] ccp-3-q3 S ffff8b5cfb300000 0 13526 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.319496] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.322126] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.327282] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.334083] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.340813] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.347531] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.352617] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.358904] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.365543] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.371814] ccp-3-q4 S ffff8b5cfb3030c0 0 13527 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.379107] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.381747] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.387715] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.394663] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.401389] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.408105] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.413190] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.419467] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.426472] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.432880] ccp-4-q0 S ffff8b5cf8e7c100 0 13534 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.440339] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.443077] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:28 2019][2937550.448569] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.453753] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.460685] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.467444] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.474155] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.479225] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.485520] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.492142] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.498421] ccp-4-q1 S ffff8b5cf8e78000 0 13539 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.505713] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.508345] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:28 2019][2937550.513845] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.519020] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.525819] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.532530] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.539253] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.544312] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.550619] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.557277] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.563562] ccp-4-q2 S ffff8b5cfaf25140 0 13540 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.570855] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.573495] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.578659] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.585475] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.592240] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.599388] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.604542] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.610824] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.617452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.623752] ccp-4-q3 S ffff8b5cfaf26180 0 13542 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.631060] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.633696] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.638859] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.645654] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.652362] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.659076] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.664149] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.670443] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.677097] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.683377] ccp-4-q4 S ffff8b5cfaf20000 0 13548 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.690675] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.693313] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.698483] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.705306] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.712034] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.718772] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.723831] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.730105] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.736727] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.742999] ccp-5-q2 S ffff8b5cfaf230c0 0 13560 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.750292] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.752925] [] ? kprobe_flush_task+0xd0/0x170 [Fri Apr 12 11:10:28 2019][2937550.759115] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.764286] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.771080] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.777786] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.784496] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.789556] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.795832] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:28 2019][2937550.802451] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.803071] LustreError: dumping log to /tmp/lustre-log.1555092628.118185 [Fri Apr 12 11:10:28 2019][2937550.815748] ccp-5-q3 S ffff8b5cfaf21040 0 13563 2 0x00000000 [Fri Apr 12 11:10:28 2019][2937550.823033] Call Trace: [Fri Apr 12 11:10:28 2019][2937550.825666] [] schedule+0x29/0x70 [Fri Apr 12 11:10:28 2019][2937550.830840] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.837651] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:28 2019][2937550.844368] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:28 2019][2937550.851094] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:28 2019][2937550.856152] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:28 2019][2937550.862429] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937550.869052] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937550.875330] ccp-5-q4 S ffff8b5cfaf24100 0 13565 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937550.882619] Call Trace: [Fri Apr 12 11:10:29 2019][2937550.885834] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937550.891739] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937550.898659] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937550.905366] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937550.912074] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937550.917733] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937550.924175] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937550.930820] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937550.937126] ccp-6-q0 S ffff8b5cfaf22080 0 13580 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937550.944834] Call Trace: [Fri Apr 12 11:10:29 2019][2937550.947469] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937550.952720] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937550.959632] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937550.966506] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937550.973244] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937550.978705] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937550.985015] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937550.991634] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937550.997906] ccp-6-q1 S ffff8b5cfa34d140 0 13582 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.005220] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.007860] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.013032] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.019858] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.026615] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.033343] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.038398] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.044674] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.051294] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.057568] ccp-6-q2 S ffff8b5cfb305140 0 13584 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.064837] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.067469] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:29 2019][2937551.072973] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.078137] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.084931] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.091641] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.098347] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.103407] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.109676] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.116294] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.122560] ccp-6-q3 S ffff8b5ceea69040 0 13601 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.129830] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.132460] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.137612] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.144405] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.151113] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.157818] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.162881] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.169156] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.175779] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.182051] ccp-6-q4 S ffff8b5ceea6b0c0 0 13603 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.189328] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.191962] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.197121] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.203934] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.210653] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.217381] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.222858] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.229325] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.235954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.242224] ccp-7-q2 S ffff8b5ceea68000 0 13776 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.249494] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.252126] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.257285] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.264099] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.270811] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.278039] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.283163] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.289436] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.296240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.302516] ccp-7-q3 S ffff8b5cee355140 0 13777 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.309789] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.312422] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.317585] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.324394] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.331129] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.337851] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.342912] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.349186] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.355804] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.362079] ccp-7-q4 S ffff8b5cee356180 0 13778 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.369383] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.372017] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.377180] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.383981] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.390752] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.397465] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.402556] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.408863] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.415664] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.421994] ccp-8-q0 S ffff8b5cee354100 0 13783 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.429298] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.431949] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.437111] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.443949] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.450679] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.457385] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.462453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.468944] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.475576] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.481857] ccp-8-q1 S ffff8b5cee3530c0 0 13784 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.489152] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.491798] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.497012] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.503810] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.511078] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.518261] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.523322] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.529623] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.536253] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.542560] ccp-8-q2 S ffff8b5cfa162080 0 13786 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.549854] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.552486] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.557650] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.564447] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.571810] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.578715] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.583794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.590067] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.596692] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.602966] ccp-8-q3 S ffff8b5cfa1630c0 0 13787 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.610242] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.612875] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.618039] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.624874] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.631611] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.638319] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.643382] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.649676] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.656297] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.662571] ccp-8-q4 S ffff8b5cfa161040 0 13788 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.669866] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.672517] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.677679] [] ccp_cmd_queue_thread+0x94/0x2b0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.684476] [] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp] [Fri Apr 12 11:10:29 2019][2937551.691185] [] ? ccp_unregister_rng+0x30/0x30 [ccp] [Fri Apr 12 11:10:29 2019][2937551.697898] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.702960] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.709305] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.715962] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.722240] kmpathd S ffff8b5cf3fc9040 0 14017 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.729543] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.732186] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.737342] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:29 2019][2937551.743191] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:29 2019][2937551.749414] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.754493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.760785] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.767427] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.773732] kmpath_handlerd S ffff8b5cf3fcc100 0 14019 2 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.781024] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.783669] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.789040] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:29 2019][2937551.794927] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:29 2019][2937551.801174] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:29 2019][2937551.806233] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.812506] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:29 2019][2937551.819146] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:29 2019][2937551.825418] multipathd S ffff8b6cf53f30c0 0 14438 1 0x00000000 [Fri Apr 12 11:10:29 2019][2937551.832696] Call Trace: [Fri Apr 12 11:10:29 2019][2937551.835337] [] schedule+0x29/0x70 [Fri Apr 12 11:10:29 2019][2937551.840494] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:29 2019][2937551.846679] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:29 2019][2937551.852174] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:30 2019][2937551.857498] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:30 2019][2937551.862643] [] ? __fput+0x186/0x260 [Fri Apr 12 11:10:30 2019][2937551.867965] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:30 2019][2937551.873289] [] ? task_work_run+0xc0/0xe0 [Fri Apr 12 11:10:30 2019][2937551.879043] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937551.885230] multipathd S ffff8b6cfc58e180 0 14477 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937551.892506] Call Trace: [Fri Apr 12 11:10:30 2019][2937551.895136] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937551.900284] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:30 2019][2937551.906472] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:30 2019][2937551.911968] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:30 2019][2937551.917287] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:30 2019][2937551.922611] [] ? fd_install+0x25/0x30 [Fri Apr 12 11:10:30 2019][2937551.928104] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937551.934294] multipathd S ffff8b5e5e4a8000 0 14512 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937551.941569] Call Trace: [Fri Apr 12 11:10:30 2019][2937551.944199] [] ? get_page_from_freelist+0x412/0xa70 [Fri Apr 12 11:10:30 2019][2937551.950908] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937551.956056] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:30 2019][2937551.963284] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:10:30 2019][2937551.968951] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:30 2019][2937551.974793] [] ? __pollwait+0x7f/0xf0 [Fri Apr 12 11:10:30 2019][2937551.980288] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:30 2019][2937551.986822] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:30 2019][2937551.993097] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:30 2019][2937551.998679] [] ? netlink_recvmsg+0x212/0x490 [Fri Apr 12 11:10:30 2019][2937552.004780] [] ? plist_add+0x91/0xd0 [Fri Apr 12 11:10:30 2019][2937552.010189] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:10:30 2019][2937552.015941] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:30 2019][2937552.021430] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:30 2019][2937552.027443] [] ? wake_up_q+0x5b/0x80 [Fri Apr 12 11:10:30 2019][2937552.032867] [] ? futex_wake_op+0x4a5/0x610 [Fri Apr 12 11:10:30 2019][2937552.038787] [] ? do_futex+0x17a/0x5a0 [Fri Apr 12 11:10:30 2019][2937552.044284] [] ? _raw_spin_unlock_bh+0x1e/0x20 [Fri Apr 12 11:10:30 2019][2937552.050559] [] ? release_sock+0x120/0x170 [Fri Apr 12 11:10:30 2019][2937552.056398] [] SyS_poll+0x74/0x110 [Fri Apr 12 11:10:30 2019][2937552.061627] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.067814] multipathd S ffff8b5e5e4a9040 0 14514 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.075089] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.077722] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:30 2019][2937552.084427] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937552.089575] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:30 2019][2937552.096709] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:30 2019][2937552.102638] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:30 2019][2937552.109952] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:30 2019][2937552.116488] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:30 2019][2937552.122760] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:30 2019][2937552.128349] [] ? get_page_from_freelist+0x412/0xa70 [Fri Apr 12 11:10:30 2019][2937552.135053] [] ? native_smp_send_reschedule+0x52/0x70 [Fri Apr 12 11:10:30 2019][2937552.141934] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:30 2019][2937552.147429] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Fri Apr 12 11:10:30 2019][2937552.154396] [] ? add_page_to_unevictable_list+0xc8/0x100 [Fri Apr 12 11:10:30 2019][2937552.161549] [] ? page_add_new_anon_rmap+0x12d/0x170 [Fri Apr 12 11:10:30 2019][2937552.168256] [] ? handle_pte_fault+0x5b8/0xd10 [Fri Apr 12 11:10:30 2019][2937552.174446] [] ? __kmalloc+0x55/0x230 [Fri Apr 12 11:10:30 2019][2937552.179940] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:30 2019][2937552.186059] [] ? __set_task_blocked+0x41/0xa0 [Fri Apr 12 11:10:30 2019][2937552.192244] [] SyS_ppoll+0xee/0x1f0 [Fri Apr 12 11:10:30 2019][2937552.197580] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.203791] multipathd S ffff8b6cf71d1040 0 16306 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.211074] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.213717] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:30 2019][2937552.220442] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937552.225595] [] do_nanosleep+0x86/0xf0 [Fri Apr 12 11:10:30 2019][2937552.231096] [] hrtimer_nanosleep+0xbb/0x180 [Fri Apr 12 11:10:30 2019][2937552.237117] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:30 2019][2937552.243063] [] ? do_nanosleep+0x72/0xf0 [Fri Apr 12 11:10:30 2019][2937552.248754] [] SyS_nanosleep+0x96/0xb0 [Fri Apr 12 11:10:30 2019][2937552.254357] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.260569] multipathd S ffff8b6cf71d0000 0 16307 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.267851] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.270493] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937552.275653] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:30 2019][2937552.281863] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:30 2019][2937552.287375] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:30 2019][2937552.292521] [] ? get_futex_key+0x1c8/0x2c0 [Fri Apr 12 11:10:30 2019][2937552.298460] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:30 2019][2937552.303782] [] ? lockref_put_or_lock+0x48/0x80 [Fri Apr 12 11:10:30 2019][2937552.310057] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:30 2019][2937552.315208] [] ? __fput+0x186/0x260 [Fri Apr 12 11:10:30 2019][2937552.320537] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:30 2019][2937552.325857] [] ? task_work_run+0xc0/0xe0 [Fri Apr 12 11:10:30 2019][2937552.331621] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.337805] multipathd S ffff8b6cfcbd30c0 0 17106 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.345098] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.347740] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937552.352907] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.359289] [] ? strlcpy+0x42/0x60 [Fri Apr 12 11:10:30 2019][2937552.364543] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:30 2019][2937552.370603] [] dev_wait+0x50/0xd0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.376536] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.382736] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Fri Apr 12 11:10:30 2019][2937552.389723] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.396176] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.402366] [] do_vfs_ioctl+0x3a0/0x5a0 [Fri Apr 12 11:10:30 2019][2937552.408035] [] ? __set_task_blocked+0x41/0xa0 [Fri Apr 12 11:10:30 2019][2937552.414236] [] ? __set_current_blocked+0x42/0x70 [Fri Apr 12 11:10:30 2019][2937552.420709] [] SyS_ioctl+0xa1/0xc0 [Fri Apr 12 11:10:30 2019][2937552.426211] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Fri Apr 12 11:10:30 2019][2937552.432549] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.438761] multipathd S ffff8b6cfcbd0000 0 17188 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.446059] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.448692] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937552.453857] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.460217] [] ? strlcpy+0x42/0x60 [Fri Apr 12 11:10:30 2019][2937552.465481] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:30 2019][2937552.471502] [] dev_wait+0x50/0xd0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.477432] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.483619] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Fri Apr 12 11:10:30 2019][2937552.490628] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.497086] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.503277] [] do_vfs_ioctl+0x3a0/0x5a0 [Fri Apr 12 11:10:30 2019][2937552.508990] [] ? __set_task_blocked+0x41/0xa0 [Fri Apr 12 11:10:30 2019][2937552.515222] [] ? __set_current_blocked+0x42/0x70 [Fri Apr 12 11:10:30 2019][2937552.522090] [] SyS_ioctl+0xa1/0xc0 [Fri Apr 12 11:10:30 2019][2937552.527891] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Fri Apr 12 11:10:30 2019][2937552.534243] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.540428] multipathd S ffff8b6cfcbd4100 0 17263 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.547702] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.550337] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937552.555502] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.561881] [] ? strlcpy+0x42/0x60 [Fri Apr 12 11:10:30 2019][2937552.567113] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:30 2019][2937552.573131] [] dev_wait+0x50/0xd0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.579058] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.585246] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Fri Apr 12 11:10:30 2019][2937552.592217] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.598662] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.604852] [] do_vfs_ioctl+0x3a0/0x5a0 [Fri Apr 12 11:10:30 2019][2937552.610518] [] ? __set_task_blocked+0x41/0xa0 [Fri Apr 12 11:10:30 2019][2937552.616703] [] ? __set_current_blocked+0x42/0x70 [Fri Apr 12 11:10:30 2019][2937552.623155] [] SyS_ioctl+0xa1/0xc0 [Fri Apr 12 11:10:30 2019][2937552.628388] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Fri Apr 12 11:10:30 2019][2937552.634674] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.640859] multipathd S ffff8b6cfcbd6180 0 17330 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.648172] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.650815] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937552.655997] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.662355] [] ? strlcpy+0x42/0x60 [Fri Apr 12 11:10:30 2019][2937552.667588] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:30 2019][2937552.673633] [] dev_wait+0x50/0xd0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.679572] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.685764] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Fri Apr 12 11:10:30 2019][2937552.692734] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.699181] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.705368] [] do_vfs_ioctl+0x3a0/0x5a0 [Fri Apr 12 11:10:30 2019][2937552.711034] [] ? __set_task_blocked+0x41/0xa0 [Fri Apr 12 11:10:30 2019][2937552.717224] [] ? __set_current_blocked+0x42/0x70 [Fri Apr 12 11:10:30 2019][2937552.723670] [] SyS_ioctl+0xa1/0xc0 [Fri Apr 12 11:10:30 2019][2937552.728907] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Fri Apr 12 11:10:30 2019][2937552.735192] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.741405] multipathd S ffff8b6cfcbd1040 0 17544 1 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.748733] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.751388] [] schedule+0x29/0x70 [Fri Apr 12 11:10:30 2019][2937552.756555] [] dm_wait_event+0xa1/0xf0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.762919] [] ? strlcpy+0x42/0x60 [Fri Apr 12 11:10:30 2019][2937552.768169] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:30 2019][2937552.774197] [] dev_wait+0x50/0xd0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.780144] [] ctl_ioctl+0x24e/0x550 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.786353] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Fri Apr 12 11:10:30 2019][2937552.793341] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.799821] [] dm_ctl_ioctl+0xe/0x20 [dm_mod] [Fri Apr 12 11:10:30 2019][2937552.806027] [] do_vfs_ioctl+0x3a0/0x5a0 [Fri Apr 12 11:10:30 2019][2937552.811695] [] ? __set_task_blocked+0x41/0xa0 [Fri Apr 12 11:10:30 2019][2937552.817879] [] ? __set_current_blocked+0x42/0x70 [Fri Apr 12 11:10:30 2019][2937552.824344] [] SyS_ioctl+0xa1/0xc0 [Fri Apr 12 11:10:30 2019][2937552.829600] [] ? SyS_rt_sigprocmask+0xc4/0x100 [Fri Apr 12 11:10:30 2019][2937552.835877] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:30 2019][2937552.842059] kvm-irqfd-clean S ffff8b6cf9656180 0 16314 2 0x00000000 [Fri Apr 12 11:10:30 2019][2937552.849335] Call Trace: [Fri Apr 12 11:10:30 2019][2937552.851970] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937552.857128] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:31 2019][2937552.862966] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:31 2019][2937552.869152] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937552.874212] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937552.880483] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937552.887151] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937552.893449] kworker/16:1H S ffff8b5cf93da080 0 16828 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937552.901093] Call Trace: [Fri Apr 12 11:10:31 2019][2937552.904243] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937552.910721] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937552.915878] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937552.921655] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937552.928418] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937552.933479] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937552.939838] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937552.946471] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937552.952745] kworker/11:1H S ffff8b8cf4458000 0 16829 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937552.960058] Call Trace: [Fri Apr 12 11:10:31 2019][2937552.962696] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937552.968881] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937552.974036] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937552.979808] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937552.986856] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937552.991957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937552.998288] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.004913] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.011195] kworker/8:1H S ffff8b5cf93de180 0 16830 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.018494] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.021138] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.027326] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.032483] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.038237] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.044944] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.050008] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.056292] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.062908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.069184] kworker/44:1H S ffff8b5cf93dc100 0 16836 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.076496] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.079128] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.085529] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.090770] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.096598] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.103343] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.108416] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.114775] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.121422] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.127723] kworker/32:1H S ffff8b5cf93db0c0 0 16838 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.135030] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.137668] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.143896] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.149080] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.154840] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.161558] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.166643] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.172944] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.179567] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.185838] kworker/34:1H S ffff8b7cf640c100 0 16839 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.193138] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.195776] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.201984] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.207148] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.212901] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.219609] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.224677] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.230964] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.237803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.244293] kworker/21:1H S ffff8b6cfb20b0c0 0 16840 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.251748] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.254393] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.260590] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.265741] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.272944] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.279842] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.284937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.291213] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.297834] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.304117] kworker/24:1H S ffff8b5cf93d9040 0 16841 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.311400] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.314036] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.320258] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.325423] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.331173] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.337881] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.342943] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.349232] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.355853] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.362124] kworker/20:1H S ffff8b5cf93dd140 0 16842 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.369416] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.372051] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.378236] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.383403] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.389155] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.395862] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.400918] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.407202] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.413821] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.420096] kworker/6:1H S ffff8b7cf6409040 0 16843 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.427399] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.430033] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.436218] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.441419] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.447193] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.453899] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.458958] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.465233] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.471857] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.478138] kworker/4:1H S ffff8b5cfa164100 0 16844 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.485432] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.488068] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.494256] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.499401] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.505156] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.511870] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.516942] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.523218] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.529840] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.536111] kworker/22:1H S ffff8b7cf640b0c0 0 16889 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.543394] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.546028] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.552215] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.557360] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.563109] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.569814] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.574868] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.581138] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.587758] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.594032] kworker/35:1H S ffff8b8cf445c100 0 16986 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.601328] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.603995] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:31 2019][2937553.610710] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.616023] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.621778] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:31 2019][2937553.628482] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.633543] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.639819] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.646464] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.652762] kdmflush S ffff8b8cf445b0c0 0 17091 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.660067] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.662713] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.667870] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.673707] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:31 2019][2937553.679894] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.685229] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.691503] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.698123] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.704664] bioset S ffff8b8cf445a080 0 17104 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.711974] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.714606] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.719754] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.725599] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:31 2019][2937553.732156] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.737676] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.744063] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.750707] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.757198] kdmwork-253:0 S ffff8b6cf71d5140 0 17105 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.764836] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.767518] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.772699] [] kthread_worker_fn+0xf7/0x180 [Fri Apr 12 11:10:31 2019][2937553.778724] [] ? kthread_stop+0xf0/0xf0 [Fri Apr 12 11:10:31 2019][2937553.784393] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.789461] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.795755] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.802378] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.808660] kdmflush S ffff8b6cf71d2080 0 17183 2 0x00000000 [Fri Apr 12 11:10:31 2019][2937553.815940] Call Trace: [Fri Apr 12 11:10:31 2019][2937553.819075] [] schedule+0x29/0x70 [Fri Apr 12 11:10:31 2019][2937553.824390] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:31 2019][2937553.830231] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:31 2019][2937553.836437] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:31 2019][2937553.841506] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.847781] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:31 2019][2937553.854401] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:31 2019][2937553.860674] bioset S ffff8b6cf71d4100 0 17185 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937553.867950] Call Trace: [Fri Apr 12 11:10:32 2019][2937553.870583] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937553.875733] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:32 2019][2937553.881573] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:32 2019][2937553.887770] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937553.892844] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937553.899156] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937553.905799] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937553.912092] kdmwork-253:1 S ffff8b6cf71d6180 0 17187 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937553.919365] Call Trace: [Fri Apr 12 11:10:32 2019][2937553.921999] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937553.927143] [] kthread_worker_fn+0xf7/0x180 [Fri Apr 12 11:10:32 2019][2937553.933157] [] ? kthread_stop+0xf0/0xf0 [Fri Apr 12 11:10:32 2019][2937553.938825] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937553.943885] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937553.950154] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937553.956774] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937553.963068] kdmflush S ffff8b6cfa3ad140 0 17258 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937553.970377] Call Trace: [Fri Apr 12 11:10:32 2019][2937553.973012] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937553.978166] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:32 2019][2937553.984009] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:32 2019][2937553.990204] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937553.995264] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.001551] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.008181] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.014470] bioset S ffff8b6cfa3ac100 0 17261 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.021796] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.024461] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.029622] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.035462] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:32 2019][2937554.041648] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.046713] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.053511] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.060459] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.066787] kdmwork-253:2 S ffff8b6cfa3ab0c0 0 17262 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.074070] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.076706] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.081870] [] kthread_worker_fn+0xf7/0x180 [Fri Apr 12 11:10:32 2019][2937554.087893] [] ? kthread_stop+0xf0/0xf0 [Fri Apr 12 11:10:32 2019][2937554.093554] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.098616] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.104893] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.111526] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.117795] kworker/13:1H S ffff8b6cfa3a9040 0 17310 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.125072] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.127703] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:32 2019][2937554.133892] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.139038] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.144793] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:32 2019][2937554.151503] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.156562] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.162837] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.169462] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.175734] kworker/19:1H S ffff8b8cf92dc100 0 17313 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.183016] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.185650] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:32 2019][2937554.191839] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.196984] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.202730] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:32 2019][2937554.209430] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.214492] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.220765] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.227377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.233645] kdmflush S ffff8b8cf92db0c0 0 17325 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.240920] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.243554] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.248700] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.254541] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:32 2019][2937554.260729] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.265792] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.272068] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.278688] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.284960] bioset S ffff8b8cf92d9040 0 17328 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.292233] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.294861] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.300008] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.305850] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:32 2019][2937554.312036] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.317099] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.323375] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.329997] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.336270] kdmwork-253:3 S ffff8b6cfa3c8000 0 17329 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.343548] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.346184] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.351340] [] kthread_worker_fn+0xf7/0x180 [Fri Apr 12 11:10:32 2019][2937554.357896] [] ? kthread_stop+0xf0/0xf0 [Fri Apr 12 11:10:32 2019][2937554.364230] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.369606] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.375879] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.382500] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.388790] kworker/12:1H S ffff8b5cfa166180 0 17445 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.396077] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.398712] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:32 2019][2937554.404907] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.410055] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.415812] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:32 2019][2937554.422562] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.427623] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.433897] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.440519] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.446793] kworker/39:1H S ffff8b8cf1ef6180 0 17527 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.454095] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.456730] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:32 2019][2937554.462931] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.468100] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.473892] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:32 2019][2937554.480927] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.486394] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.492783] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.499401] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.505672] kdmflush S ffff8b8cf92dd140 0 17539 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.512957] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.515592] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.520761] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.526608] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:32 2019][2937554.532802] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.537861] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.544142] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.550757] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.557031] bioset S ffff8b8cf92d8000 0 17542 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.564308] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.566940] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.572087] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.577928] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:32 2019][2937554.584116] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.589791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.596393] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.603046] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.609319] kdmwork-253:4 S ffff8b8cf92de180 0 17543 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.616597] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.619227] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.624373] [] kthread_worker_fn+0xf7/0x180 [Fri Apr 12 11:10:32 2019][2937554.630388] [] ? kthread_stop+0xf0/0xf0 [Fri Apr 12 11:10:32 2019][2937554.636057] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.641137] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.647411] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.654060] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.660343] kworker/17:1H S ffff8b6cf53f4100 0 17602 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.667633] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.670271] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:32 2019][2937554.676467] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.681621] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.687378] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:32 2019][2937554.694093] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.699151] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.705429] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.712068] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.718342] kworker/1:1H S ffff8b6cf53f6180 0 17603 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.725661] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.728644] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:32 2019][2937554.735039] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.740194] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:32 2019][2937554.745943] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:32 2019][2937554.752644] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:32 2019][2937554.757703] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.763972] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:32 2019][2937554.770629] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:32 2019][2937554.776911] kworker/45:1H S ffff8b6cf53f5140 0 17624 2 0x00000000 [Fri Apr 12 11:10:32 2019][2937554.784207] Call Trace: [Fri Apr 12 11:10:32 2019][2937554.786846] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:32 2019][2937554.793036] [] schedule+0x29/0x70 [Fri Apr 12 11:10:32 2019][2937554.798190] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:33 2019][2937554.804034] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:33 2019][2937554.810742] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:33 2019][2937554.815801] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:33 2019][2937554.822076] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:33 2019][2937554.828698] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:33 2019][2937554.834972] kworker/36:1H S ffff8b5cfa160000 0 17647 2 0x00000000 [Fri Apr 12 11:10:33 2019][2937554.842271] Call Trace: [Fri Apr 12 11:10:33 2019][2937554.844907] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:33 2019][2937554.851102] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937554.856761] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:33 2019][2937554.862795] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:33 2019][2937554.869617] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:33 2019][2937554.874686] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:33 2019][2937554.880977] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:33 2019][2937554.887634] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:33 2019][2937554.893907] auditd S ffff8b6cf68230c0 0 17978 1 0x00000000 [Fri Apr 12 11:10:33 2019][2937554.901218] Call Trace: [Fri Apr 12 11:10:33 2019][2937554.903862] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:33 2019][2937554.910585] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937554.915748] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:33 2019][2937554.922920] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:33 2019][2937554.928869] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:33 2019][2937554.936194] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:33 2019][2937554.942768] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:33 2019][2937554.948005] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:33 2019][2937554.953759] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:33 2019][2937554.959516] [] ? SyS_write+0xa4/0xf0 [Fri Apr 12 11:10:33 2019][2937554.964923] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:33 2019][2937554.971107] auditd S ffff8b8cf548a080 0 17979 1 0x00000000 [Fri Apr 12 11:10:33 2019][2937554.978378] Call Trace: [Fri Apr 12 11:10:33 2019][2937554.981009] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937554.986159] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:33 2019][2937554.992348] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:33 2019][2937554.997852] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:10:33 2019][2937555.003528] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:33 2019][2937555.008869] [] ? jbd2_complete_transaction+0x52/0xa0 [jbd2] [Fri Apr 12 11:10:33 2019][2937555.016264] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:33 2019][2937555.021587] [] ? do_fsync+0x78/0xb0 [Fri Apr 12 11:10:33 2019][2937555.026912] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:33 2019][2937555.033105] rpciod S ffff8b8cf6b64100 0 17983 2 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.040386] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.043023] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937555.048168] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:33 2019][2937555.054013] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:33 2019][2937555.060197] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:33 2019][2937555.065273] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:33 2019][2937555.071550] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:33 2019][2937555.078254] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:33 2019][2937555.084627] xprtiod S ffff8b8cf6b62080 0 17984 2 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.091904] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.094538] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937555.099683] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:33 2019][2937555.105525] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:33 2019][2937555.111712] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:33 2019][2937555.116774] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:33 2019][2937555.123050] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:33 2019][2937555.129668] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:33 2019][2937555.135935] irqbalance S ffff8b6cfa3cc100 0 18015 1 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.143203] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.145834] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:33 2019][2937555.152541] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937555.157689] [] do_nanosleep+0x86/0xf0 [Fri Apr 12 11:10:33 2019][2937555.163187] [] hrtimer_nanosleep+0xbb/0x180 [Fri Apr 12 11:10:33 2019][2937555.169201] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:33 2019][2937555.175149] [] ? do_nanosleep+0x72/0xf0 [Fri Apr 12 11:10:33 2019][2937555.180813] [] SyS_nanosleep+0x96/0xb0 [Fri Apr 12 11:10:33 2019][2937555.186396] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:33 2019][2937555.192581] polkitd S ffff8b6cf53f0000 0 18019 1 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.199850] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.202480] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937555.207628] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:33 2019][2937555.214857] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:33 2019][2937555.220696] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:33 2019][2937555.227244] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:33 2019][2937555.233546] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:33 2019][2937555.239124] [] ? get_futex_key_refs.isra.12+0x33/0x50 [Fri Apr 12 11:10:33 2019][2937555.246007] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:10:33 2019][2937555.251759] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:33 2019][2937555.257255] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:33 2019][2937555.262749] [] ? ttwu_do_activate+0x6f/0x80 [Fri Apr 12 11:10:33 2019][2937555.268764] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:33 2019][2937555.274780] [] ? inotify_free_event+0xe/0x10 [Fri Apr 12 11:10:33 2019][2937555.280881] [] ? kfree+0x106/0x140 [Fri Apr 12 11:10:33 2019][2937555.286116] [] ? list_del+0xd/0x30 [Fri Apr 12 11:10:33 2019][2937555.291364] [] ? inotify_read+0x2ef/0x420 [Fri Apr 12 11:10:33 2019][2937555.297200] [] ? eventfd_ctx_read+0x67/0x1e0 [Fri Apr 12 11:10:33 2019][2937555.303303] [] SyS_poll+0x74/0x110 [Fri Apr 12 11:10:33 2019][2937555.308538] [] ? __audit_syscall_exit+0x1e6/0x280 [Fri Apr 12 11:10:33 2019][2937555.315071] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:33 2019][2937555.321258] gmain S ffff8b7cd825a080 0 18046 1 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.328527] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.331158] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937555.336303] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:33 2019][2937555.343526] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:33 2019][2937555.349379] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:33 2019][2937555.355909] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:33 2019][2937555.362204] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:33 2019][2937555.367828] [] ? radix_tree_lookup_slot+0x22/0x50 [Fri Apr 12 11:10:33 2019][2937555.374375] [] ? __find_get_page+0x1e/0xa0 [Fri Apr 12 11:10:33 2019][2937555.380297] [] ? filemap_fault+0x17e/0x490 [Fri Apr 12 11:10:33 2019][2937555.386217] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:33 2019][2937555.391710] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:33 2019][2937555.397200] [] ? do_read_fault.isra.61+0x139/0x1b0 [Fri Apr 12 11:10:33 2019][2937555.403819] [] ? handle_pte_fault+0x2f4/0xd10 [Fri Apr 12 11:10:33 2019][2937555.410022] [] ? eventfd_ctx_read+0x67/0x1e0 [Fri Apr 12 11:10:33 2019][2937555.416120] [] SyS_poll+0x74/0x110 [Fri Apr 12 11:10:33 2019][2937555.421354] [] ? __audit_syscall_exit+0x1e6/0x280 [Fri Apr 12 11:10:33 2019][2937555.427887] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:33 2019][2937555.434078] gdbus S ffff8b6cf88a4100 0 18047 1 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.441440] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.444072] [] ? __kmalloc_reserve.isra.32+0x31/0x90 [Fri Apr 12 11:10:33 2019][2937555.450903] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937555.456048] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:33 2019][2937555.463289] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:33 2019][2937555.469146] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:33 2019][2937555.475003] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:33 2019][2937555.481549] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:33 2019][2937555.487865] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:33 2019][2937555.493449] [] ? sock_sendmsg+0xb6/0xf0 [Fri Apr 12 11:10:33 2019][2937555.499140] [] ? sock_recvmsg+0xc5/0x100 [Fri Apr 12 11:10:33 2019][2937555.504896] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:10:33 2019][2937555.510660] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:33 2019][2937555.516163] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:33 2019][2937555.521656] [] ? ___sys_sendmsg+0x3e9/0x400 [Fri Apr 12 11:10:33 2019][2937555.527674] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:33 2019][2937555.534303] [] ? _raw_spin_lock+0x20/0x30 [Fri Apr 12 11:10:33 2019][2937555.540426] [] ? futex_wake+0x90/0x180 [Fri Apr 12 11:10:33 2019][2937555.546012] [] ? eventfd_ctx_read+0x67/0x1e0 [Fri Apr 12 11:10:33 2019][2937555.552115] [] ? __sys_sendmsg+0x85/0x90 [Fri Apr 12 11:10:33 2019][2937555.557867] [] SyS_poll+0x74/0x110 [Fri Apr 12 11:10:33 2019][2937555.563101] [] ? __audit_syscall_exit+0x1e6/0x280 [Fri Apr 12 11:10:33 2019][2937555.569637] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:33 2019][2937555.575849] polkitd S ffff8b6cf7ef1040 0 18048 1 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.583164] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.585812] [] ? __rmqueue+0x8a/0x460 [Fri Apr 12 11:10:33 2019][2937555.591325] [] ? get_page_from_freelist+0x412/0xa70 [Fri Apr 12 11:10:33 2019][2937555.598054] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937555.603276] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:33 2019][2937555.610511] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:33 2019][2937555.616351] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:33 2019][2937555.622886] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:33 2019][2937555.629170] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:33 2019][2937555.634763] [] ? __alloc_pages_nodemask+0x176/0x420 [Fri Apr 12 11:10:33 2019][2937555.641794] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:33 2019][2937555.647342] [] ? physflat_send_IPI_mask+0xe/0x10 [Fri Apr 12 11:10:33 2019][2937555.653830] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:33 2019][2937555.659845] [] ? wake_up_q+0x5b/0x80 [Fri Apr 12 11:10:33 2019][2937555.665279] [] ? futex_wake+0x90/0x180 [Fri Apr 12 11:10:33 2019][2937555.670892] [] ? eventfd_ctx_read+0x67/0x1e0 [Fri Apr 12 11:10:33 2019][2937555.677003] [] SyS_poll+0x74/0x110 [Fri Apr 12 11:10:33 2019][2937555.682249] [] ? __audit_syscall_exit+0x1e6/0x280 [Fri Apr 12 11:10:33 2019][2937555.688784] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:33 2019][2937555.694994] JS GC Helper S ffff8b6cf7ef2080 0 18049 1 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.702287] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.704930] [] schedule+0x29/0x70 [Fri Apr 12 11:10:33 2019][2937555.710075] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:33 2019][2937555.716317] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:33 2019][2937555.721844] [] ? handle_pte_fault+0x2f4/0xd10 [Fri Apr 12 11:10:33 2019][2937555.728032] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:33 2019][2937555.733364] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:33 2019][2937555.738702] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:33 2019][2937555.745372] JS Sour~ Thread S ffff8b6cf7ef6180 0 18051 1 0x00000080 [Fri Apr 12 11:10:33 2019][2937555.752823] Call Trace: [Fri Apr 12 11:10:33 2019][2937555.755457] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937555.760668] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:34 2019][2937555.766896] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:34 2019][2937555.772387] [] ? get_futex_key+0x1c8/0x2c0 [Fri Apr 12 11:10:34 2019][2937555.778328] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:34 2019][2937555.784352] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:34 2019][2937555.789691] [] ? call_rwsem_wake+0x1b/0x30 [Fri Apr 12 11:10:34 2019][2937555.795628] [] ? yama_task_prctl+0x24/0x110 [Fri Apr 12 11:10:34 2019][2937555.801646] [] ? strlcpy+0x42/0x60 [Fri Apr 12 11:10:34 2019][2937555.806885] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:34 2019][2937555.812208] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937555.818391] polkitd S ffff8b6cf53230c0 0 18057 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937555.825677] Call Trace: [Fri Apr 12 11:10:34 2019][2937555.828309] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937555.833458] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:34 2019][2937555.840685] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:34 2019][2937555.846613] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:34 2019][2937555.852456] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:34 2019][2937555.859013] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:34 2019][2937555.865291] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:34 2019][2937555.870874] [] ? __alloc_pages_nodemask+0x176/0x420 [Fri Apr 12 11:10:34 2019][2937555.877579] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937555.883085] [] ? physflat_send_IPI_mask+0xe/0x10 [Fri Apr 12 11:10:34 2019][2937555.889561] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:34 2019][2937555.895572] [] ? wake_up_q+0x5b/0x80 [Fri Apr 12 11:10:34 2019][2937555.900983] [] ? futex_wake+0x90/0x180 [Fri Apr 12 11:10:34 2019][2937555.906584] [] ? eventfd_ctx_read+0x67/0x1e0 [Fri Apr 12 11:10:34 2019][2937555.912711] [] SyS_poll+0x74/0x110 [Fri Apr 12 11:10:34 2019][2937555.917953] [] ? __audit_syscall_exit+0x1e6/0x280 [Fri Apr 12 11:10:34 2019][2937555.923211] LustreError: dumping log to /tmp/lustre-log.1555092634.118329 [Fri Apr 12 11:10:34 2019][2937555.931442] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937555.937632] rpcbind S ffff8b8cf7299040 0 18022 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937555.944906] Call Trace: [Fri Apr 12 11:10:34 2019][2937555.947542] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:34 2019][2937555.954246] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937555.959394] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:34 2019][2937555.966537] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:34 2019][2937555.972465] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:34 2019][2937555.979780] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:34 2019][2937555.986319] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:34 2019][2937555.992597] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:34 2019][2937555.998182] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:34 2019][2937556.003672] [] ? local_clock+0x25/0x30 [Fri Apr 12 11:10:34 2019][2937556.009256] [] ? calc_timer_values+0x1c/0x90 [Fri Apr 12 11:10:34 2019][2937556.015395] [] ? ktime_get_mono_fast_ns+0x48/0x90 [Fri Apr 12 11:10:34 2019][2937556.021950] [] ? watchdog_overflow_callback+0x3a/0x140 [Fri Apr 12 11:10:34 2019][2937556.028939] [] ? __perf_event_overflow+0x57/0x100 [Fri Apr 12 11:10:34 2019][2937556.035474] [] ? perf_event_overflow+0x14/0x20 [Fri Apr 12 11:10:34 2019][2937556.041749] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.047243] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.052735] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.058230] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.063726] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.069220] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.074723] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.080216] [] ? ktime_get_ts64+0x52/0xf0 [Fri Apr 12 11:10:34 2019][2937556.086057] [] SyS_poll+0x74/0x110 [Fri Apr 12 11:10:34 2019][2937556.091285] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.097472] smartd S ffff8b6cf53f1040 0 18024 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.104749] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.107380] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:34 2019][2937556.114090] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.119256] [] do_nanosleep+0x86/0xf0 [Fri Apr 12 11:10:34 2019][2937556.124763] [] hrtimer_nanosleep+0xbb/0x180 [Fri Apr 12 11:10:34 2019][2937556.130803] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:34 2019][2937556.136749] [] ? do_nanosleep+0x72/0xf0 [Fri Apr 12 11:10:34 2019][2937556.142412] [] SyS_nanosleep+0x96/0xb0 [Fri Apr 12 11:10:34 2019][2937556.147995] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.154230] dbus-daemon S ffff8b6cf53f2080 0 18028 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.161517] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.164150] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.169295] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:34 2019][2937556.176527] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Fri Apr 12 11:10:34 2019][2937556.183490] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:34 2019][2937556.190031] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:34 2019][2937556.195274] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:34 2019][2937556.201047] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:34 2019][2937556.206801] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.213005] dsm_sa_eventmgr S ffff8b8cf5ed1040 0 18032 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.220297] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.222928] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.228076] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:34 2019][2937556.235324] [] ? tlb_finish_mmu+0x55/0x60 [Fri Apr 12 11:10:34 2019][2937556.241182] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:34 2019][2937556.247291] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:34 2019][2937556.253826] [] do_sigtimedwait+0x196/0x240 [Fri Apr 12 11:10:34 2019][2937556.259752] [] SYSC_rt_sigtimedwait+0xb4/0x120 [Fri Apr 12 11:10:34 2019][2937556.266025] [] ? __do_page_fault+0x228/0x500 [Fri Apr 12 11:10:34 2019][2937556.272126] [] SyS_rt_sigtimedwait+0xe/0x10 [Fri Apr 12 11:10:34 2019][2937556.278141] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.284329] dsm_sa_eventmgr S ffff8b5cee350000 0 18040 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.291627] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.294325] [] ? security_ipc_permission+0x1f/0x30 [Fri Apr 12 11:10:34 2019][2937556.300965] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.306113] [] SYSC_semtimedop+0x89a/0xc90 [Fri Apr 12 11:10:34 2019][2937556.312042] [] ? futex_wake+0x90/0x180 [Fri Apr 12 11:10:34 2019][2937556.317623] [] ? do_smart_update+0x11d/0x130 [Fri Apr 12 11:10:34 2019][2937556.323746] [] SyS_semtimedop+0xe/0x10 [Fri Apr 12 11:10:34 2019][2937556.329332] [] SyS_semop+0x10/0x20 [Fri Apr 12 11:10:34 2019][2937556.334585] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.340791] dsm_sa_eventmgr S ffff8b5cf8786180 0 19169 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.348081] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.350712] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.355860] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:34 2019][2937556.363100] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:34 2019][2937556.368940] [] ? __pollwait+0x7f/0xf0 [Fri Apr 12 11:10:34 2019][2937556.374435] [] ? list_del+0xd/0x30 [Fri Apr 12 11:10:34 2019][2937556.379679] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:34 2019][2937556.386214] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:34 2019][2937556.392504] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:10:34 2019][2937556.397922] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.403417] [] ? __enqueue_entity+0x78/0x80 [Fri Apr 12 11:10:34 2019][2937556.409461] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:10:34 2019][2937556.415219] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:10:34 2019][2937556.421881] [] ? probe_sched_wakeup+0x2b/0xa0 [Fri Apr 12 11:10:34 2019][2937556.428122] [] ? ttwu_do_wakeup+0xb5/0xe0 [Fri Apr 12 11:10:34 2019][2937556.433972] [] ? ttwu_do_activate+0x6f/0x80 [Fri Apr 12 11:10:34 2019][2937556.440016] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:34 2019][2937556.446073] [] ? wake_up_process+0x15/0x20 [Fri Apr 12 11:10:34 2019][2937556.452065] [] ? wake_up_sem_queue_do+0x37/0x60 [Fri Apr 12 11:10:34 2019][2937556.458441] [] ? SYSC_semtimedop+0x254/0xc90 [Fri Apr 12 11:10:34 2019][2937556.464562] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:10:34 2019][2937556.470528] [] ? SYSC_recvfrom+0x127/0x160 [Fri Apr 12 11:10:34 2019][2937556.476461] [] SyS_select+0xca/0x120 [Fri Apr 12 11:10:34 2019][2937556.481869] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.488063] sssd S ffff8b5e5e4d30c0 0 18059 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.495339] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.497974] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:34 2019][2937556.504690] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.509838] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:34 2019][2937556.516980] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:34 2019][2937556.522904] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:34 2019][2937556.530257] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:34 2019][2937556.536792] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:34 2019][2937556.543065] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:34 2019][2937556.548652] [] ? __mem_cgroup_uncharge_common+0xcd/0x2f0 [Fri Apr 12 11:10:34 2019][2937556.555797] [] ? lru_cache_add+0xe/0x10 [Fri Apr 12 11:10:34 2019][2937556.561463] [] ? putback_lru_page+0x52/0xb0 [Fri Apr 12 11:10:34 2019][2937556.567477] [] ? migrate_pages+0x2d7/0x790 [Fri Apr 12 11:10:34 2019][2937556.573405] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.578921] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.584505] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.590075] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.595656] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.601153] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.606661] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.612157] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:34 2019][2937556.617654] [] ? ktime_get_ts64+0x52/0xf0 [Fri Apr 12 11:10:34 2019][2937556.623494] [] SyS_poll+0x74/0x110 [Fri Apr 12 11:10:34 2019][2937556.628738] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.634931] mdadm S ffff8b8cf7298000 0 18063 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.642217] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.644855] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:34 2019][2937556.651610] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.656756] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:34 2019][2937556.663900] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:34 2019][2937556.669822] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:34 2019][2937556.677149] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:34 2019][2937556.683704] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:34 2019][2937556.689995] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:10:34 2019][2937556.695443] [] ? __alloc_pages_nodemask+0x176/0x420 [Fri Apr 12 11:10:34 2019][2937556.702165] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Fri Apr 12 11:10:34 2019][2937556.709140] [] ? lru_cache_add+0xe/0x10 [Fri Apr 12 11:10:34 2019][2937556.714816] [] ? free_pcppages_bulk+0x17e/0x3a0 [Fri Apr 12 11:10:34 2019][2937556.721196] [] ? free_hot_cold_page+0x101/0x160 [Fri Apr 12 11:10:34 2019][2937556.727620] [] ? free_hot_cold_page_list+0x46/0xa0 [Fri Apr 12 11:10:34 2019][2937556.734307] [] ? release_pages+0x24e/0x430 [Fri Apr 12 11:10:34 2019][2937556.740351] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:10:34 2019][2937556.746313] [] ? evict+0x115/0x180 [Fri Apr 12 11:10:34 2019][2937556.751559] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:10:34 2019][2937556.757404] [] ? d_free+0x4f/0x70 [Fri Apr 12 11:10:34 2019][2937556.762551] [] ? __dentry_kill+0x128/0x180 [Fri Apr 12 11:10:34 2019][2937556.768511] [] ? ktime_get_ts64+0x52/0xf0 [Fri Apr 12 11:10:34 2019][2937556.774377] [] SyS_select+0xca/0x120 [Fri Apr 12 11:10:34 2019][2937556.779787] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.785980] gssproxy S ffff8b7cfebc6180 0 18066 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.793281] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.795925] [] ? __d_alloc+0x25/0x180 [Fri Apr 12 11:10:34 2019][2937556.801424] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.806581] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:34 2019][2937556.813818] [] ? __d_alloc+0x25/0x180 [Fri Apr 12 11:10:34 2019][2937556.819312] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:34 2019][2937556.825849] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:34 2019][2937556.831084] [] ? SyS_epoll_ctl+0x130/0xc80 [Fri Apr 12 11:10:34 2019][2937556.837010] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:34 2019][2937556.842764] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:34 2019][2937556.848525] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:34 2019][2937556.854716] gssproxy S ffff8b8cf5488000 0 18070 1 0x00000080 [Fri Apr 12 11:10:34 2019][2937556.862010] Call Trace: [Fri Apr 12 11:10:34 2019][2937556.864649] [] schedule+0x29/0x70 [Fri Apr 12 11:10:34 2019][2937556.869813] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:35 2019][2937556.876079] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:35 2019][2937556.881638] [] ? handle_pte_fault+0x2f4/0xd10 [Fri Apr 12 11:10:35 2019][2937556.887923] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:35 2019][2937556.893341] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:35 2019][2937556.898702] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937556.904972] gssproxy S ffff8b8cf548d140 0 18071 1 0x00000080 [Fri Apr 12 11:10:35 2019][2937556.912373] Call Trace: [Fri Apr 12 11:10:35 2019][2937556.915021] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937556.920172] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:35 2019][2937556.926368] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:35 2019][2937556.931861] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:35 2019][2937556.937193] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:35 2019][2937556.942512] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937556.948721] gssproxy S ffff8b8cf548e180 0 18073 1 0x00000080 [Fri Apr 12 11:10:35 2019][2937556.955992] Call Trace: [Fri Apr 12 11:10:35 2019][2937556.958624] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937556.963791] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:35 2019][2937556.969997] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:35 2019][2937556.975499] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:35 2019][2937556.980819] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:35 2019][2937556.986163] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937556.992362] gssproxy S ffff8b8cf548c100 0 18074 1 0x00000080 [Fri Apr 12 11:10:35 2019][2937556.999647] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.002286] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.007437] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:35 2019][2937557.013652] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:35 2019][2937557.019149] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:35 2019][2937557.025420] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:10:35 2019][2937557.031351] [] ? idle_balance+0x7f/0x250 [Fri Apr 12 11:10:35 2019][2937557.037103] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:35 2019][2937557.042453] [] ? schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.047770] [] ? do_exit+0x6bb/0xa40 [Fri Apr 12 11:10:35 2019][2937557.053195] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:35 2019][2937557.058521] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937557.064705] gssproxy S ffff8b8cfa7130c0 0 18075 1 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.072000] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.074641] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.079785] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:35 2019][2937557.085973] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:35 2019][2937557.091468] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:35 2019][2937557.097741] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:10:35 2019][2937557.103674] [] ? idle_balance+0x7f/0x250 [Fri Apr 12 11:10:35 2019][2937557.109439] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:35 2019][2937557.114761] [] ? schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.120084] [] ? do_exit+0x6bb/0xa40 [Fri Apr 12 11:10:35 2019][2937557.125514] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:35 2019][2937557.130837] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937557.137050] chronyd S ffff8b7cfebc4100 0 18083 1 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.144330] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.146967] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.153732] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.158878] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:35 2019][2937557.166023] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:35 2019][2937557.171950] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:35 2019][2937557.179263] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:35 2019][2937557.185799] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:35 2019][2937557.192072] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:10:35 2019][2937557.197485] [] ? udp_rmem_release+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.203495] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:35 2019][2937557.208988] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:35 2019][2937557.214482] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:35 2019][2937557.219970] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:35 2019][2937557.225487] [] ? sock_sendmsg+0xb6/0xf0 [Fri Apr 12 11:10:35 2019][2937557.231183] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:10:35 2019][2937557.236854] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:10:35 2019][2937557.242783] [] ? kmem_cache_free+0x1e2/0x200 [Fri Apr 12 11:10:35 2019][2937557.248883] [] ? __d_free+0x35/0x40 [Fri Apr 12 11:10:35 2019][2937557.254205] [] ? d_free+0x60/0x70 [Fri Apr 12 11:10:35 2019][2937557.259353] [] ? __dentry_kill+0x128/0x180 [Fri Apr 12 11:10:35 2019][2937557.265296] [] ? ktime_get_ts64+0x52/0xf0 [Fri Apr 12 11:10:35 2019][2937557.271134] [] SyS_select+0xca/0x120 [Fri Apr 12 11:10:35 2019][2937557.276543] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937557.282725] sssd_be S ffff8b8cf5ed0000 0 18103 18059 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.289994] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.292626] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.299333] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.304480] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:35 2019][2937557.311622] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:35 2019][2937557.317550] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:35 2019][2937557.324865] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:35 2019][2937557.331402] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:35 2019][2937557.336639] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:35 2019][2937557.342758] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:35 2019][2937557.348530] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:35 2019][2937557.354283] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937557.360469] sssd_nss S ffff8b5cf7fb1040 0 18105 18059 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.367744] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.370378] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.377083] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.382237] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:35 2019][2937557.389375] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:35 2019][2937557.395305] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:35 2019][2937557.402618] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:35 2019][2937557.409156] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:35 2019][2937557.414393] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:35 2019][2937557.419554] [] ? __fput+0x186/0x260 [Fri Apr 12 11:10:35 2019][2937557.424879] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:35 2019][2937557.430640] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:35 2019][2937557.436395] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937557.442577] sssd_pam S ffff8b5cf7fb0000 0 18106 18059 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.449846] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.452479] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.459186] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.464333] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:35 2019][2937557.471474] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:35 2019][2937557.477403] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:35 2019][2937557.484718] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:35 2019][2937557.491255] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:35 2019][2937557.496518] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:35 2019][2937557.502626] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:35 2019][2937557.508379] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:35 2019][2937557.514145] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937557.520330] systemd-logind S ffff8b6cf88a30c0 0 18110 1 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.527631] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.530263] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.535408] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:35 2019][2937557.542630] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Fri Apr 12 11:10:35 2019][2937557.549599] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:35 2019][2937557.556134] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:35 2019][2937557.561380] [] ? SyS_epoll_ctl+0x24d/0xc80 [Fri Apr 12 11:10:35 2019][2937557.567335] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:35 2019][2937557.573114] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:35 2019][2937557.578888] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:35 2019][2937557.585073] mlx4 S ffff8b6cfa3aa080 0 18210 2 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.592362] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.594998] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.600816] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.606780] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:35 2019][2937557.612982] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:35 2019][2937557.618064] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.624346] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:35 2019][2937557.631439] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.638160] mlx4_ib S ffff8b6cfa3ae180 0 18223 2 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.645497] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.648149] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.653302] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.659142] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:35 2019][2937557.665332] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:35 2019][2937557.670401] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.676684] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:35 2019][2937557.683305] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.689597] mlx4_ib_mcg S ffff8b6cfa3a8000 0 18224 2 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.697500] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.700209] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.705373] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.711222] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:35 2019][2937557.717404] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:35 2019][2937557.722467] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.728744] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:35 2019][2937557.735365] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.741639] ipoib_flush S ffff8b5e5e4ad140 0 18354 2 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.748920] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.751555] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.756701] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.762542] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:35 2019][2937557.768731] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:35 2019][2937557.773791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.780089] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:35 2019][2937557.786722] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.793028] mlx5i S ffff8b5e5e4ab0c0 0 18356 2 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.800306] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.802943] [] schedule+0x29/0x70 [Fri Apr 12 11:10:35 2019][2937557.808087] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:35 2019][2937557.813928] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:35 2019][2937557.820116] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:35 2019][2937557.825178] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.831459] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:35 2019][2937557.838580] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:35 2019][2937557.845369] kworker/37:1H S ffff8b5e5e4ac100 0 18358 2 0x00000080 [Fri Apr 12 11:10:35 2019][2937557.852699] Call Trace: [Fri Apr 12 11:10:35 2019][2937557.855334] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:36 2019][2937557.861522] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937557.866669] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:36 2019][2937557.873095] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:36 2019][2937557.880175] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:36 2019][2937557.885261] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:36 2019][2937557.891563] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:36 2019][2937557.898190] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:36 2019][2937557.904484] ipoib_wq S ffff8b5e5e4aa080 0 18359 2 0x00000080 [Fri Apr 12 11:10:36 2019][2937557.911774] Call Trace: [Fri Apr 12 11:10:36 2019][2937557.914427] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937557.919592] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:36 2019][2937557.925440] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:36 2019][2937557.931628] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:36 2019][2937557.936698] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:36 2019][2937557.942980] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:36 2019][2937557.949603] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:36 2019][2937557.955897] rdma_cm S ffff8b7cf3d26180 0 18728 2 0x00000080 [Fri Apr 12 11:10:36 2019][2937557.963184] Call Trace: [Fri Apr 12 11:10:36 2019][2937557.965831] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937557.970984] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:36 2019][2937557.976821] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:36 2019][2937557.983014] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:36 2019][2937557.988075] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:36 2019][2937557.994346] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:36 2019][2937558.000968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:36 2019][2937558.007240] dsm_sa_datamgrd S ffff8b8cfa716180 0 18871 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.014511] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.017141] [] ? select_task_rq_fair+0x5a6/0x760 [Fri Apr 12 11:10:36 2019][2937558.023589] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.028737] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:36 2019][2937558.035969] [] ? physflat_send_IPI_mask+0xe/0x10 [Fri Apr 12 11:10:36 2019][2937558.042413] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:36 2019][2937558.048429] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:36 2019][2937558.054529] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:36 2019][2937558.061064] [] do_sigtimedwait+0x196/0x240 [Fri Apr 12 11:10:36 2019][2937558.066992] [] SYSC_rt_sigtimedwait+0xb4/0x120 [Fri Apr 12 11:10:36 2019][2937558.073267] [] ? call_rwsem_wake+0x1b/0x30 [Fri Apr 12 11:10:36 2019][2937558.079196] [] ? remove_vma+0x66/0x70 [Fri Apr 12 11:10:36 2019][2937558.084690] [] SyS_rt_sigtimedwait+0xe/0x10 [Fri Apr 12 11:10:36 2019][2937558.090705] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.096892] dsm_sa_datamgrd S ffff8b6cf9b3a080 0 19149 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.104169] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.106805] [] ? security_ipc_permission+0x1f/0x30 [Fri Apr 12 11:10:36 2019][2937558.113425] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.118570] [] SYSC_semtimedop+0x89a/0xc90 [Fri Apr 12 11:10:36 2019][2937558.124498] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:36 2019][2937558.130511] [] ? wake_up_q+0x5b/0x80 [Fri Apr 12 11:10:36 2019][2937558.135941] [] ? rwsem_wake+0x8b/0xe0 [Fri Apr 12 11:10:36 2019][2937558.141445] [] ? call_rwsem_wake+0x1b/0x30 [Fri Apr 12 11:10:36 2019][2937558.147413] [] ? up_read+0x1f/0x30 [Fri Apr 12 11:10:36 2019][2937558.152650] [] ? __do_page_fault+0x228/0x500 [Fri Apr 12 11:10:36 2019][2937558.158762] [] SyS_semtimedop+0xe/0x10 [Fri Apr 12 11:10:36 2019][2937558.164354] [] SyS_semop+0x10/0x20 [Fri Apr 12 11:10:36 2019][2937558.169605] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.175790] dsm_sa_datamgrd S ffff8b6cf9b38000 0 19150 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.183090] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.185735] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.190890] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:36 2019][2937558.198136] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:36 2019][2937558.204004] [] ? __pollwait+0x7f/0xf0 [Fri Apr 12 11:10:36 2019][2937558.209538] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:36 2019][2937558.216074] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:36 2019][2937558.222346] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:10:36 2019][2937558.227756] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:36 2019][2937558.233248] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:36 2019][2937558.238746] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:36 2019][2937558.244250] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:36 2019][2937558.249746] [] ? list_del+0xd/0x30 [Fri Apr 12 11:10:36 2019][2937558.254979] [] ? __rmqueue+0x8a/0x460 [Fri Apr 12 11:10:36 2019][2937558.260494] [] ? zone_statistics+0x88/0xa0 [Fri Apr 12 11:10:36 2019][2937558.266419] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:10:36 2019][2937558.272923] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:36 2019][2937558.278570] [] ? physflat_send_IPI_mask+0xe/0x10 [Fri Apr 12 11:10:36 2019][2937558.285036] [] ? native_smp_send_reschedule+0x52/0x70 [Fri Apr 12 11:10:36 2019][2937558.291918] [] ? resched_curr+0xae/0xc0 [Fri Apr 12 11:10:36 2019][2937558.297612] [] ? check_preempt_curr+0x80/0xa0 [Fri Apr 12 11:10:36 2019][2937558.303808] [] ? wake_up_new_task+0x131/0x1a0 [Fri Apr 12 11:10:36 2019][2937558.309993] [] SyS_select+0xca/0x120 [Fri Apr 12 11:10:36 2019][2937558.315404] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.321588] dsm_sa_datamgrd S ffff8b6cf9b3b0c0 0 19151 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.328876] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.331523] [] ? security_ipc_permission+0x1f/0x30 [Fri Apr 12 11:10:36 2019][2937558.338162] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.343312] [] SYSC_semtimedop+0x89a/0xc90 [Fri Apr 12 11:10:36 2019][2937558.349266] [] ? SYSC_sendto+0x17e/0x1c0 [Fri Apr 12 11:10:36 2019][2937558.355021] [] SyS_semtimedop+0xe/0x10 [Fri Apr 12 11:10:36 2019][2937558.360611] [] SyS_semop+0x10/0x20 [Fri Apr 12 11:10:36 2019][2937558.365849] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.372054] dsm_sa_datamgrd S ffff8b6cf9b3d140 0 19152 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.379381] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.382039] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:36 2019][2937558.388780] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.393925] [] do_nanosleep+0x86/0xf0 [Fri Apr 12 11:10:36 2019][2937558.399424] [] hrtimer_nanosleep+0xbb/0x180 [Fri Apr 12 11:10:36 2019][2937558.405441] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:36 2019][2937558.411380] [] ? do_nanosleep+0x72/0xf0 [Fri Apr 12 11:10:36 2019][2937558.417052] [] SyS_nanosleep+0x96/0xb0 [Fri Apr 12 11:10:36 2019][2937558.422633] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.428825] dsm_sa_datamgrd S ffff8b6cf9b3c100 0 19153 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.436102] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.438733] [] ? security_ipc_permission+0x1f/0x30 [Fri Apr 12 11:10:36 2019][2937558.445352] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.450503] [] SYSC_semtimedop+0x89a/0xc90 [Fri Apr 12 11:10:36 2019][2937558.456430] [] SyS_semtimedop+0xe/0x10 [Fri Apr 12 11:10:36 2019][2937558.462009] [] SyS_semop+0x10/0x20 [Fri Apr 12 11:10:36 2019][2937558.467246] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.473435] dsm_sa_datamgrd S ffff8b5e5e4d2080 0 19154 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.480736] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.483379] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:36 2019][2937558.490105] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.495278] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:36 2019][2937558.502441] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:36 2019][2937558.508375] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:36 2019][2937558.515684] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:36 2019][2937558.522225] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:36 2019][2937558.528516] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:10:36 2019][2937558.533932] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:36 2019][2937558.539453] [] ? security_inode_permission+0x22/0x30 [Fri Apr 12 11:10:36 2019][2937558.546256] [] ? __inode_permission+0x52/0xd0 [Fri Apr 12 11:10:36 2019][2937558.552444] [] ? inode_permission+0x18/0x50 [Fri Apr 12 11:10:36 2019][2937558.558495] [] ? link_path_walk+0x27e/0x8b0 [Fri Apr 12 11:10:36 2019][2937558.564520] [] ? __d_lookup+0x7a/0x160 [Fri Apr 12 11:10:36 2019][2937558.570097] [] ? __d_instantiate+0x2d/0xe0 [Fri Apr 12 11:10:36 2019][2937558.576025] [] ? security_inode_permission+0x22/0x30 [Fri Apr 12 11:10:36 2019][2937558.582848] [] ? __inode_permission+0x52/0xd0 [Fri Apr 12 11:10:36 2019][2937558.589034] [] ? inode_permission+0x18/0x50 [Fri Apr 12 11:10:36 2019][2937558.595050] [] ? link_path_walk+0x27e/0x8b0 [Fri Apr 12 11:10:36 2019][2937558.601063] [] ? __d_lookup+0x7a/0x160 [Fri Apr 12 11:10:36 2019][2937558.606642] [] ? lookup_fast+0x12d/0x230 [Fri Apr 12 11:10:36 2019][2937558.612402] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:36 2019][2937558.618106] [] ? path_lookupat+0x122/0x8b0 [Fri Apr 12 11:10:36 2019][2937558.624132] [] ? kmem_cache_alloc+0x35/0x1f0 [Fri Apr 12 11:10:36 2019][2937558.630247] [] ? getname_flags+0x4f/0x1a0 [Fri Apr 12 11:10:36 2019][2937558.636088] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:10:36 2019][2937558.642018] [] ? cp_new_stat+0x165/0x1a0 [Fri Apr 12 11:10:36 2019][2937558.647782] [] ? ktime_get_ts64+0x52/0xf0 [Fri Apr 12 11:10:36 2019][2937558.653633] [] SyS_select+0xca/0x120 [Fri Apr 12 11:10:36 2019][2937558.659073] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.665278] dsm_sa_datamgrd S ffff8b5e5e4d6180 0 19157 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.672549] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.675196] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:36 2019][2937558.681911] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.687718] [] do_nanosleep+0x86/0xf0 [Fri Apr 12 11:10:36 2019][2937558.693396] [] hrtimer_nanosleep+0xbb/0x180 [Fri Apr 12 11:10:36 2019][2937558.699420] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:36 2019][2937558.705353] [] ? do_nanosleep+0x72/0xf0 [Fri Apr 12 11:10:36 2019][2937558.711056] [] SyS_nanosleep+0x96/0xb0 [Fri Apr 12 11:10:36 2019][2937558.716645] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.722850] dsm_sa_datamgrd S ffff8b8cf92a30c0 0 19168 1 0x00000080 [Fri Apr 12 11:10:36 2019][2937558.730145] Call Trace: [Fri Apr 12 11:10:36 2019][2937558.732794] [] ? hrtimer_try_to_cancel.part.26+0x58/0x100 [Fri Apr 12 11:10:36 2019][2937558.740026] [] schedule+0x29/0x70 [Fri Apr 12 11:10:36 2019][2937558.745177] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:36 2019][2937558.752427] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:36 2019][2937558.758275] [] ? __pollwait+0x7f/0xf0 [Fri Apr 12 11:10:36 2019][2937558.763775] [] ? list_del+0xd/0x30 [Fri Apr 12 11:10:36 2019][2937558.769019] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:36 2019][2937558.775563] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:36 2019][2937558.781857] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:10:36 2019][2937558.787262] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:36 2019][2937558.792758] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:36 2019][2937558.799206] [] ? pollwake+0x73/0x90 [Fri Apr 12 11:10:36 2019][2937558.804537] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:36 2019][2937558.810288] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:36 2019][2937558.816305] [] ? __wake_up_sync_key+0x4f/0x60 [Fri Apr 12 11:10:36 2019][2937558.822494] [] ? sock_def_readable+0x3a/0x70 [Fri Apr 12 11:10:36 2019][2937558.828593] [] ? unix_stream_sendmsg+0x3fd/0x430 [Fri Apr 12 11:10:36 2019][2937558.835071] [] ? sock_sendmsg+0xb6/0xf0 [Fri Apr 12 11:10:36 2019][2937558.840747] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:10:36 2019][2937558.846673] [] ? SYSC_sendto+0x17e/0x1c0 [Fri Apr 12 11:10:36 2019][2937558.852439] [] SyS_select+0xca/0x120 [Fri Apr 12 11:10:36 2019][2937558.857858] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:36 2019][2937558.864377] kworker/5:1H S ffff8b6cfacc8000 0 18978 2 0x00000080 [Fri Apr 12 11:10:37 2019][2937558.872522] Call Trace: [Fri Apr 12 11:10:37 2019][2937558.875170] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:37 2019][2937558.881358] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937558.886507] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:37 2019][2937558.892273] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:37 2019][2937558.899624] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:37 2019][2937558.904896] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:37 2019][2937558.911172] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:37 2019][2937558.917790] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:37 2019][2937558.924066] collectd S ffff8b6cfc4f6180 0 19282 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937558.931344] Call Trace: [Fri Apr 12 11:10:37 2019][2937558.933975] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:37 2019][2937558.940683] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937558.945827] [] do_nanosleep+0x86/0xf0 [Fri Apr 12 11:10:37 2019][2937558.951319] [] hrtimer_nanosleep+0xbb/0x180 [Fri Apr 12 11:10:37 2019][2937558.957385] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:37 2019][2937558.963313] [] ? do_nanosleep+0x72/0xf0 [Fri Apr 12 11:10:37 2019][2937558.968982] [] SyS_nanosleep+0x96/0xb0 [Fri Apr 12 11:10:37 2019][2937558.974561] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937558.980748] writer#0 S ffff8b7cfebd0000 0 19291 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937558.988051] Call Trace: [Fri Apr 12 11:10:37 2019][2937558.990720] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937558.995868] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.002055] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.007559] [] ? physflat_send_IPI_mask+0xe/0x10 [Fri Apr 12 11:10:37 2019][2937559.014005] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:37 2019][2937559.020023] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.025379] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:37 2019][2937559.030609] [] ? __fput+0x186/0x260 [Fri Apr 12 11:10:37 2019][2937559.036076] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.041405] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.047598] writer#1 S ffff8b7cfebd5140 0 19292 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.054962] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.057604] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.062768] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.068953] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.074452] [] ? physflat_send_IPI_mask+0xe/0x10 [Fri Apr 12 11:10:37 2019][2937559.080921] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:37 2019][2937559.087007] [] ? get_futex_key+0x1c8/0x2c0 [Fri Apr 12 11:10:37 2019][2937559.092937] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.098255] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:37 2019][2937559.103402] [] ? __fput+0x186/0x260 [Fri Apr 12 11:10:37 2019][2937559.108728] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.114077] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.120281] writer#2 S ffff8b7cfebd6180 0 19293 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.127578] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.130211] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.135360] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.141556] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.147050] [] ? physflat_send_IPI_mask+0xe/0x10 [Fri Apr 12 11:10:37 2019][2937559.153508] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:37 2019][2937559.159541] [] ? get_futex_key+0x1c8/0x2c0 [Fri Apr 12 11:10:37 2019][2937559.165994] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.171991] [] ? __switch_to+0x151/0x580 [Fri Apr 12 11:10:37 2019][2937559.177840] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.183211] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.189425] writer#3 S ffff8b7cfebd4100 0 19294 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.196730] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.199366] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.204516] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.210707] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.216813] [] ? ttwu_do_wakeup+0xb5/0xe0 [Fri Apr 12 11:10:37 2019][2937559.222893] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:37 2019][2937559.228912] [] ? get_futex_key+0x1c8/0x2c0 [Fri Apr 12 11:10:37 2019][2937559.234837] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.240168] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.245510] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.251703] writer#4 S ffff8b7cfebd1040 0 19295 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.258995] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.261626] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.266775] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.272960] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.278459] [] ? ttwu_do_wakeup+0xb5/0xe0 [Fri Apr 12 11:10:37 2019][2937559.284297] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:37 2019][2937559.290311] [] ? get_futex_key+0x1c8/0x2c0 [Fri Apr 12 11:10:37 2019][2937559.296240] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.301566] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:37 2019][2937559.306712] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.312036] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.318237] reader#0 S ffff8b7cd825b0c0 0 19296 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.325513] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.328150] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:37 2019][2937559.334863] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.340031] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.346220] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.351740] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:37 2019][2937559.357693] [] ? futex_wait_queue_me+0xa4/0x130 [Fri Apr 12 11:10:37 2019][2937559.364054] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.369384] [] ? do_munmap+0x327/0x480 [Fri Apr 12 11:10:37 2019][2937559.374963] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.380287] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.386480] reader#1 S ffff8b7cd825d140 0 19297 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.393766] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.396407] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:37 2019][2937559.403143] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.408288] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.414484] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.419979] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:37 2019][2937559.425904] [] ? futex_wait_queue_me+0xa4/0x130 [Fri Apr 12 11:10:37 2019][2937559.432271] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.437612] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:37 2019][2937559.442784] [] ? __fput+0x186/0x260 [Fri Apr 12 11:10:37 2019][2937559.448123] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.453445] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.459627] reader#2 S ffff8b7cd825e180 0 19298 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.467145] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.470321] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:37 2019][2937559.477330] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.482488] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.488682] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.494188] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:37 2019][2937559.500116] [] ? futex_wait_queue_me+0xa4/0x130 [Fri Apr 12 11:10:37 2019][2937559.507226] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.512906] [] ? do_munmap+0x327/0x480 [Fri Apr 12 11:10:37 2019][2937559.518624] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.523957] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.530141] reader#3 S ffff8b7cd8259040 0 19299 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.537495] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.540140] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:37 2019][2937559.546883] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.552039] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.558237] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.563743] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:37 2019][2937559.569689] [] ? futex_wait_queue_me+0xa4/0x130 [Fri Apr 12 11:10:37 2019][2937559.576052] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.581383] [] ? do_munmap+0x327/0x480 [Fri Apr 12 11:10:37 2019][2937559.586985] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.592310] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.598495] reader#4 S ffff8b7cd8258000 0 19300 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.605774] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.608404] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.613550] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:37 2019][2937559.619739] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:37 2019][2937559.625242] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:37 2019][2937559.631520] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:10:37 2019][2937559.637447] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:10:37 2019][2937559.643201] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:37 2019][2937559.648534] [] ? schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.653876] [] ? do_exit+0x6bb/0xa40 [Fri Apr 12 11:10:37 2019][2937559.659288] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:37 2019][2937559.664628] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.670813] rsyslogd S ffff8b6cfc4f5140 0 19283 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.678128] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.680773] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:37 2019][2937559.687490] [] schedule+0x29/0x70 [Fri Apr 12 11:10:37 2019][2937559.693229] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:37 2019][2937559.700707] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:37 2019][2937559.706646] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:37 2019][2937559.713971] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:37 2019][2937559.720532] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:37 2019][2937559.727230] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:10:37 2019][2937559.732897] [] ? zone_statistics+0x88/0xa0 [Fri Apr 12 11:10:37 2019][2937559.738842] [] ? __inc_zone_page_state+0x32/0x60 [Fri Apr 12 11:10:37 2019][2937559.745295] [] ? do_page_add_anon_rmap+0x9a/0xc0 [Fri Apr 12 11:10:37 2019][2937559.751766] [] ? task_scan_min+0x3e/0x60 [Fri Apr 12 11:10:37 2019][2937559.757526] [] ? task_scan_max+0x27/0x40 [Fri Apr 12 11:10:37 2019][2937559.763299] [] ? task_numa_fault+0x8d0/0xbb0 [Fri Apr 12 11:10:37 2019][2937559.769415] [] ? do_numa_page+0x1be/0x250 [Fri Apr 12 11:10:37 2019][2937559.775257] [] ? handle_pte_fault+0x316/0xd10 [Fri Apr 12 11:10:37 2019][2937559.781449] [] ? putback_lru_page+0x52/0xb0 [Fri Apr 12 11:10:37 2019][2937559.787485] [] ? migrate_pages+0x2d7/0x790 [Fri Apr 12 11:10:37 2019][2937559.793420] [] ? should_numa_migrate_memory+0x5b/0x150 [Fri Apr 12 11:10:37 2019][2937559.800420] [] ? mpol_misplaced+0x18c/0x1e0 [Fri Apr 12 11:10:37 2019][2937559.806437] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:10:37 2019][2937559.812373] [] ? handle_mm_fault+0x39d/0x9b0 [Fri Apr 12 11:10:37 2019][2937559.818498] [] ? __check_object_size+0x1ca/0x250 [Fri Apr 12 11:10:37 2019][2937559.824948] [] ? ktime_get_ts64+0x52/0xf0 [Fri Apr 12 11:10:37 2019][2937559.830809] [] SyS_select+0xca/0x120 [Fri Apr 12 11:10:37 2019][2937559.836226] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:37 2019][2937559.842409] in:imjournal S ffff8b7cfede2080 0 19305 1 0x00000080 [Fri Apr 12 11:10:37 2019][2937559.849705] Call Trace: [Fri Apr 12 11:10:37 2019][2937559.852354] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:38 2019][2937559.859063] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937559.864231] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:38 2019][2937559.871386] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:38 2019][2937559.877309] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:38 2019][2937559.884636] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:38 2019][2937559.891189] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:38 2019][2937559.898064] [] do_sys_poll+0x48d/0x590 [Fri Apr 12 11:10:38 2019][2937559.904313] [] ? jbd2_journal_dirty_metadata+0xd1/0x260 [jbd2] [Fri Apr 12 11:10:38 2019][2937559.912075] [] ? jbd2_journal_release_jbd_inode+0xf1/0x130 [jbd2] [Fri Apr 12 11:10:38 2019][2937559.920053] [] ? security_inode_permission+0x22/0x30 [Fri Apr 12 11:10:38 2019][2937559.926855] [] ? __inode_permission+0x52/0xd0 [Fri Apr 12 11:10:38 2019][2937559.933066] [] ? inode_permission+0x18/0x50 [Fri Apr 12 11:10:38 2019][2937559.939082] [] ? link_path_walk+0x81/0x8b0 [Fri Apr 12 11:10:38 2019][2937559.945017] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:38 2019][2937559.950513] [] ? complete_walk+0x50/0xe0 [Fri Apr 12 11:10:38 2019][2937559.956275] [] ? ttwu_do_wakeup+0xb5/0xe0 [Fri Apr 12 11:10:38 2019][2937559.962120] [] ? ttwu_do_activate+0x6f/0x80 [Fri Apr 12 11:10:38 2019][2937559.968143] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:10:38 2019][2937559.974153] [] ? kmem_cache_alloc+0x35/0x1f0 [Fri Apr 12 11:10:38 2019][2937559.980266] [] ? wake_up_q+0x5b/0x80 [Fri Apr 12 11:10:38 2019][2937559.985693] [] ? futex_wake_op+0x4a5/0x610 [Fri Apr 12 11:10:38 2019][2937559.991621] [] ? do_futex+0x17a/0x5a0 [Fri Apr 12 11:10:38 2019][2937559.997113] [] ? ktime_get_ts64+0x52/0xf0 [Fri Apr 12 11:10:38 2019][2937560.002956] [] SyS_ppoll+0x1d3/0x1f0 [Fri Apr 12 11:10:38 2019][2937560.008364] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:38 2019][2937560.014580] rs:main Q:Reg S ffff8b5cf6ba2080 0 19307 1 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.021897] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.024541] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.029722] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:38 2019][2937560.035934] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:38 2019][2937560.041439] [] ? generic_file_aio_write+0x77/0xa0 [Fri Apr 12 11:10:38 2019][2937560.049324] [] ? ext4_file_write+0xd2/0x1e0 [ext4] [Fri Apr 12 11:10:38 2019][2937560.056390] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:38 2019][2937560.061711] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:38 2019][2937560.067035] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:38 2019][2937560.073269] sshd S ffff8b6cfb77c100 0 19290 1 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.080581] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.083232] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.088404] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:38 2019][2937560.095631] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:10:38 2019][2937560.101471] [] ? __pollwait+0x7f/0xf0 [Fri Apr 12 11:10:38 2019][2937560.106968] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:38 2019][2937560.113530] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:10:38 2019][2937560.119805] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:10:38 2019][2937560.125222] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:38 2019][2937560.130723] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:10:38 2019][2937560.136230] [] ? __alloc_skb+0xa1/0x2d0 [Fri Apr 12 11:10:38 2019][2937560.141912] [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 [Fri Apr 12 11:10:38 2019][2937560.148879] [] ? cpumask_any_but+0x39/0x50 [Fri Apr 12 11:10:38 2019][2937560.154830] [] ? flush_tlb_page+0x38/0xa0 [Fri Apr 12 11:10:38 2019][2937560.160687] [] ? lru_cache_add+0xe/0x10 [Fri Apr 12 11:10:38 2019][2937560.166371] [] ? page_add_new_anon_rmap+0xb8/0x170 [Fri Apr 12 11:10:38 2019][2937560.173010] [] ? wp_page_copy.isra.71+0x3ab/0x560 [Fri Apr 12 11:10:38 2019][2937560.179547] [] ? unlock_page+0x2b/0x30 [Fri Apr 12 11:10:38 2019][2937560.185780] [] ? do_wp_page+0x19e/0x720 [Fri Apr 12 11:10:38 2019][2937560.191597] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:10:38 2019][2937560.197561] [] ? handle_mm_fault+0x39d/0x9b0 [Fri Apr 12 11:10:38 2019][2937560.203679] [] SyS_select+0xca/0x120 [Fri Apr 12 11:10:38 2019][2937560.209087] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:38 2019][2937560.215272] nfsiod S ffff8b5cfb638000 0 19365 2 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.222557] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.225189] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.230341] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:38 2019][2937560.236213] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:38 2019][2937560.242403] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:38 2019][2937560.247493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:38 2019][2937560.253803] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:38 2019][2937560.260420] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:38 2019][2937560.266712] kworker/18:1H S ffff8b7cf7ff9040 0 19432 2 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.273996] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.276629] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:38 2019][2937560.282815] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.287965] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:38 2019][2937560.293731] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:38 2019][2937560.300455] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:38 2019][2937560.305545] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:38 2019][2937560.311829] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:38 2019][2937560.318445] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:38 2019][2937560.324721] kworker/2:1H S ffff8b7cf7ffc100 0 19433 2 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.332001] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.334638] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:38 2019][2937560.340824] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.345983] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:38 2019][2937560.351736] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:38 2019][2937560.358443] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:38 2019][2937560.363507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:38 2019][2937560.369801] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:38 2019][2937560.376418] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:38 2019][2937560.382702] nfsv4.1-svc S ffff8b7cf7ffb0c0 0 19435 2 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.390957] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.393626] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.398802] [] nfs41_callback_svc+0x15d/0x1c0 [nfsv4] [Fri Apr 12 11:10:38 2019][2937560.405718] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:38 2019][2937560.411751] [] ? nfs4_callback_svc+0x60/0x60 [nfsv4] [Fri Apr 12 11:10:38 2019][2937560.418586] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:38 2019][2937560.423676] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:38 2019][2937560.429957] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:38 2019][2937560.436593] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:38 2019][2937560.442870] crond S ffffffffa456796e 0 19481 1 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.450167] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.452815] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:38 2019][2937560.459597] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.464812] [] do_nanosleep+0x86/0xf0timer_nanosleep+0xbb/0x180 [Fri Apr 12 11:10:38 2019][2937560.476320] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:38 2019][2937560.482253] [] ? do_nanosleep+0x72/0xf0 [Fri Apr 12 11:10:38 2019][2937560.487948] [] SyS_nanosleep+0x96/0xb0 [Fri Apr 12 11:10:38 2019][2937560.493543] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:38 2019][2937560.499729] agetty S ffff8b6cf9b3e180 0 19552 1 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.507043] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.509696] [] ? bit_cursor+0x663/0x6a0 [Fri Apr 12 11:10:38 2019][2937560.515377] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.520523] [] schedule_timeout+0x221/0x2d0 [Fri Apr 12 11:10:38 2019][2937560.526557] [] ? get_work_pool+0x3e/0x70 [Fri Apr 12 11:10:38 2019][2937560.532342] [] ? flush_work+0x13b/0x190 [Fri Apr 12 11:10:38 2019][2937560.538015] [] ? set_room+0x68/0xc0 [Fri Apr 12 11:10:38 2019][2937560.543333] [] n_tty_read+0x27c/0x930 [Fri Apr 12 11:10:38 2019][2937560.548841] [] ? ldsem_down_read+0x3b/0x220 [Fri Apr 12 11:10:38 2019][2937560.554892] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:38 2019][2937560.560651] [] tty_read+0x90/0x100 [Fri Apr 12 11:10:38 2019][2937560.565904] [] vfs_read+0x9f/0x170 [Fri Apr 12 11:10:38 2019][2937560.571138] [] SyS_read+0x7f/0xf0 [Fri Apr 12 11:10:38 2019][2937560.576285] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:38 2019][2937560.582470] agetty S ffff8b7cf3241040 0 19555 1 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.589747] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.592380] [] ? __do_fault.isra.59+0x8a/0x100 [Fri Apr 12 11:10:38 2019][2937560.598709] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.603871] [] schedule_timeout+0x221/0x2d0 [Fri Apr 12 11:10:38 2019][2937560.609897] [] ? get_work_pool+0x3e/0x70 [Fri Apr 12 11:10:38 2019][2937560.615649] [] ? flush_work+0x13b/0x190 [Fri Apr 12 11:10:38 2019][2937560.621319] [] ? do_read_fault.isra.61+0x139/0x1b0 [Fri Apr 12 11:10:38 2019][2937560.627957] [] ? serial8250_start_tx+0xa5/0xe0 [Fri Apr 12 11:10:38 2019][2937560.634277] [] ? set_room+0x68/0xc0 [Fri Apr 12 11:10:38 2019][2937560.639597] [] n_tty_read+0x27c/0x930 [Fri Apr 12 11:10:38 2019][2937560.645100] [] ? ldsem_down_read+0x3b/0x220 [Fri Apr 12 11:10:38 2019][2937560.651126] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:38 2019][2937560.656879] [] tty_read+0x90/0x100 [Fri Apr 12 11:10:38 2019][2937560.662123] [] vfs_read+0x9f/0x170 [Fri Apr 12 11:10:38 2019][2937560.667355] [] SyS_read+0x7f/0xf0 [Fri Apr 12 11:10:38 2019][2937560.673114] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:38 2019][2937560.679631] master S ffff8b8cf31fa080 0 19681 1 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.686948] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.689581] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:38 2019][2937560.696285] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.701461] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:38 2019][2937560.708622] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:38 2019][2937560.714553] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:38 2019][2937560.721899] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:38 2019][2937560.728451] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:38 2019][2937560.733684] [] ? hrtimer_start+0x12/0x20 [Fri Apr 12 11:10:38 2019][2937560.739439] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:38 2019][2937560.745196] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:38 2019][2937560.750952] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:38 2019][2937560.757134] slapd S ffff8b7cfe58b0c0 0 19691 1 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.764410] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.767044] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.772193] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:38 2019][2937560.778382] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:38 2019][2937560.784322] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:10:38 2019][2937560.790471] [] ? sched_slice.isra.58+0x66/0xc0 [Fri Apr 12 11:10:38 2019][2937560.796767] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:38 2019][2937560.802093] [] ? resched_curr+0xae/0xc0 [Fri Apr 12 11:10:38 2019][2937560.807769] [] ? check_preempt_curr+0x80/0xa0 [Fri Apr 12 11:10:38 2019][2937560.814004] [] ? wake_up_new_task+0x131/0x1a0 [Fri Apr 12 11:10:38 2019][2937560.820239] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:38 2019][2937560.825566] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:38 2019][2937560.831747] slapd S ffff8b8cfa5f0000 0 19732 1 0x00000080 [Fri Apr 12 11:10:38 2019][2937560.839024] Call Trace: [Fri Apr 12 11:10:38 2019][2937560.841657] [] ? pipe_read+0x2e5/0x400 [Fri Apr 12 11:10:38 2019][2937560.847237] [] schedule+0x29/0x70 [Fri Apr 12 11:10:38 2019][2937560.852384] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:10:38 2019][2937560.859613] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [Fri Apr 12 11:10:38 2019][2937560.866580] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:39 2019][2937560.873144] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:39 2019][2937560.878377] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:39 2019][2937560.884129] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:39 2019][2937560.889886] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:39 2019][2937560.896070] slapd S ffff8b5cf6cbb0c0 0 19733 1 0x00000080 [Fri Apr 12 11:10:39 2019][2937560.903341] Call Trace: [Fri Apr 12 11:10:39 2019][2937560.905972] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937560.911120] [] futex_wait_queue_me+0xc6/0x130 [Fri Apr 12 11:10:39 2019][2937560.917308] [] futex_wait+0x17b/0x280 [Fri Apr 12 11:10:39 2019][2937560.922802] [] ? __sb_end_write+0x31/0x60 [Fri Apr 12 11:10:39 2019][2937560.928642] [] ? pipe_write+0x48f/0x540 [Fri Apr 12 11:10:39 2019][2937560.934311] [] do_futex+0x106/0x5a0 [Fri Apr 12 11:10:39 2019][2937560.939633] [] SyS_futex+0x80/0x190 [Fri Apr 12 11:10:39 2019][2937560.944956] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:39 2019][2937560.951143] qmgr S ffff8b5cf6cbe180 0 19693 19681 0x00000080 [Fri Apr 12 11:10:39 2019][2937560.958418] Call Trace: [Fri Apr 12 11:10:39 2019][2937560.961050] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:39 2019][2937560.967758] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937560.972906] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:39 2019][2937560.980047] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:39 2019][2937560.985974] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:39 2019][2937560.993287] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:39 2019][2937560.999817] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:39 2019][2937561.005052] [] ? mntput+0x24/0x40 [Fri Apr 12 11:10:39 2019][2937561.010198] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:39 2019][2937561.015954] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:39 2019][2937561.021708] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:39 2019][2937561.027893] kworker/38:1H S ffff8b7cf9b4a080 0 19844 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.035169] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.037804] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.043993] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.044001] LustreError: dumping log to /tmp/lustre-log.1555092639.118764 [Fri Apr 12 11:10:39 2019][2937561.056098] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.061858] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.068562] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.073626] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.079930] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.086557] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.092828] kworker/40:1H S ffff8b5cf5d91040 0 20078 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.100123] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.102757] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.108942] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.114088] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.119844] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.126553] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.131624] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.137915] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.144539] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.150837] kworker/33:1H S ffff8b5e5e4d5140 0 20100 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.158137] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.160780] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.166971] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.172831] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.179069] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.185891] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.190949] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.197241] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.203871] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.210154] kworker/26:1H S ffff8b7cf32b5140 0 20543 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.217436] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.220071] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.226263] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.231440] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.237221] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.243929] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.248987] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.255272] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.261922] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.268205] kworker/9:1H S ffff8b5e5e48e180 0 20548 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.275490] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.278132] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.284315] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.289463] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.295217] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.301926] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.307001] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.313819] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.320504] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.326774] kworker/7:1H S ffff8b8cf8b26180 0 20551 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.334064] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.336722] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.342917] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.348061] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.353813] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.360541] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.365613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.371898] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.378528] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.384796] kworker/3:1H S ffff8b8cf8b22080 0 20552 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.392092] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.394730] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.400921] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.406067] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.411823] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.418531] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.423591] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.429867] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.436493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.442762] kworker/25:1H S ffff8b6cf33f0000 0 20553 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.450046] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.452678] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.458867] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.464013] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.469779] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.476513] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.481582] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.487859] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.494477] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.500751] kworker/27:1H S ffff8b8cf8b21040 0 20554 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.508054] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.510696] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.516883] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.522032] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.527784] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.534492] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.539553] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.545832] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.552453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.558732] kworker/14:1H S ffff8b7cf32b0000 0 20662 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.566833] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.569468] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.575672] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.580854] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.586627] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.593361] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.598421] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.604699] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.611316] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.617626] kworker/42:1H S ffff8b7cfc302080 0 20877 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.624906] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.627543] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.633730] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.639174] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.645026] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.651764] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.656826] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.663114] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.669749] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.676023] kworker/10:1H S ffff8b7cfc306180 0 20883 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.683313] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.685952] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.692138] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.697283] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.703035] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.709736] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.714799] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.721072] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.727705] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.733979] kworker/23:1H S ffff8b8cff015140 0 20961 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.741285] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.743923] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.750120] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.755292] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.761047] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.767754] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.772814] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.779088] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.785741] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.792013] kworker/43:1H S ffff8b8cf6b65140 0 21060 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.799562] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.802287] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.808478] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.813632] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:39 2019][2937561.819402] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:39 2019][2937561.826108] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:39 2019][2937561.831167] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.837444] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:39 2019][2937561.844067] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:39 2019][2937561.850338] kworker/46:1H S ffff8b7cf3031040 0 21098 2 0x00000080 [Fri Apr 12 11:10:39 2019][2937561.857674] Call Trace: [Fri Apr 12 11:10:39 2019][2937561.860309] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:39 2019][2937561.866506] [] schedule+0x29/0x70 [Fri Apr 12 11:10:39 2019][2937561.871672] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937561.877425] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937561.884151] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937561.889221] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937561.895515] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937561.902169] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937561.908442] kworker/41:1H S ffff8b6cf74cc100 0 21302 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937561.916869] Call Trace: [Fri Apr 12 11:10:40 2019][2937561.919568] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937561.925965] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937561.931200] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937561.936976] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937561.943683] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937561.948755] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937561.955058] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937561.961704] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937561.967985] kworker/31:1H S ffff8b8cf6b630c0 0 21319 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937561.975301] Call Trace: [Fri Apr 12 11:10:40 2019][2937561.977935] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937561.984143] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937561.989300] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937561.995061] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.001768] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.007441] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.013882] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.020525] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.026795] kworker/29:1H S ffff8b6cf90d4100 0 21320 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.034097] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.036736] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.042980] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.048141] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.053903] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.060616] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.065687] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.071961] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.078587] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.084875] kworker/30:1H S ffff8b7cf3302080 0 21344 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.093036] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.095891] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.102079] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.107238] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.112993] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.119702] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.124769] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.131049] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.137677] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.144005] kworker/28:1H S ffff8b5cfbdbd140 0 21363 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.151342] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.153996] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.160222] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.165417] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.171183] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.177912] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.182987] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.189272] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.195917] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.202205] kworker/47:1H S ffff8b8cf6b61040 0 21549 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.209523] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.212168] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.218354] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.223503] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.229269] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.235975] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.241034] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.247308] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.253929] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.260201] kworker/7:1 S ffff8b8b872cd140 0 39505 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.267477] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.270111] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.276299] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.281448] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.287201] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.293909] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.298969] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.305246] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.311865] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.318143] kworker/13:1 S ffff8b6ce6b80000 0 85666 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.325440] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.328077] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.334935] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.340295] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.346048] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.352758] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.357819] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.364109] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.370778] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.377464] kworker/30:2 S ffff8b76fa3e1040 0 54410 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.384775] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.387430] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.393615] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.398781] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.404533] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.411241] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.416302] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.422577] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.429197] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.435474] kworker/45:2 S ffff8b6c162930c0 0 115197 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.442842] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.445491] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.451692] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.456850] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.462606] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.469886] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.475131] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.481402] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.488021] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.494299] kworker/6:2 S ffff8b7ba4384100 0 91769 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.501579] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.504215] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.510411] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.515568] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.521326] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.528035] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.533095] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.539363] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.545982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.552251] kworker/10:2 S ffff8b791671a080 0 118970 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.559620] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.562256] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.568442] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.573588] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.579345] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.586051] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.591112] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.597386] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.604010] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.610297] kworker/41:2 S ffff8b6b8e03c100 0 19904 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.617574] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.620210] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.626453] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.631610] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.637363] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.644069] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.649127] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.655398] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.662030] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.668701] kworker/11:0 S ffff8b8cf4459040 0 46320 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.676694] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.679358] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.685571] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.690734] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.696489] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.703193] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.708252] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.714522] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.721144] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.727416] kworker/25:2 S ffff8b67cbe59040 0 85417 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.734718] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.737358] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.743577] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.748722] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.754475] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.761182] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.766247] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.773291] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.780009] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.786821] kworker/9:2 S ffff8b6cfb20e180 0 87683 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.794183] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.796829] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.803032] [] schedule+0x29/0x70 [Fri Apr 12 11:10:40 2019][2937562.808176] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:40 2019][2937562.813931] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:40 2019][2937562.820637] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:40 2019][2937562.825697] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.831978] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:40 2019][2937562.838607] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:40 2019][2937562.844879] kworker/19:1 S ffff8b88d036c100 0 4033 2 0x00000080 [Fri Apr 12 11:10:40 2019][2937562.852181] Call Trace: [Fri Apr 12 11:10:40 2019][2937562.854814] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:40 2019][2937562.861020] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937562.866163] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937562.871922] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937562.878629] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937562.883688] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937562.889986] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937562.896616] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937562.902889] kworker/15:1 R running task 0 8937 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937562.910170] Call Trace: [Fri Apr 12 11:10:41 2019][2937562.912803] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937562.919006] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937562.924156] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937562.929949] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937562.936683] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937562.941739] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937562.948019] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937562.955295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937562.961825] kworker/29:1 S ffff8b6cfb20c100 0 42240 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937562.969130] Call Trace: [Fri Apr 12 11:10:41 2019][2937562.971768] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937562.977981] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937562.983127] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937562.988879] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937562.995586] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.000660] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.006936] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.013583] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.019852] kworker/43:1 S ffff8b867b7d4100 0 58393 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.027175] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.029807] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.036013] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.041158] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.046915] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.053624] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.058681] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.064969] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.072149] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.078819] kworker/33:2 S ffff8b6cfc4ba080 0 62438 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.086142] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.088779] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.093933] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.099697] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.106414] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.111491] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.117777] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.124395] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.130696] kworker/30:1 S ffff8b7748779040 0 100096 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.138076] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.140721] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.145884] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.151653] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.158369] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.163446] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.169724] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.176371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.182655] kworker/5:1 S ffff8b6b4ef16180 0 13443 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.189952] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.192638] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.198828] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.203973] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.209728] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.216470] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.221546] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.228388] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.235314] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.241621] kworker/23:1 S ffff8b86da3ce180 0 23386 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.248922] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.251585] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.257769] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.262916] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.268671] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.275378] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.280438] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.286707] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.293330] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.299603] kworker/12:0 S ffff8b51eb175140 0 24475 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.306893] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.309529] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.314679] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.320450] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.327159] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.332219] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.338514] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.345134] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.351404] kworker/21:0 S ffff8b6c010ad140 0 27581 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.358698] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.361333] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.367541] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.372710] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.378464] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.385173] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.390852] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.397537] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.404214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.410499] python S ffff8b8bc622b0c0 0 39148 1 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.417771] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.420407] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.427120] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.432294] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:10:41 2019][2937563.439465] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:10:41 2019][2937563.445461] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:10:41 2019][2937563.452778] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:10:41 2019][2937563.459309] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:10:41 2019][2937563.464553] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:41 2019][2937563.470325] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:10:41 2019][2937563.476142] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:10:41 2019][2937563.482330] kworker/u264:0 S ffff8b774877d140 0 103155 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.489722] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.492383] [] ? ib_cq_poll_work+0x48/0x70 [ib_core] [Fri Apr 12 11:10:41 2019][2937563.499178] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.505365] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.510515] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.516266] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.523021] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.528085] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.534375] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.541013] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.547283] kworker/2:0 S ffff8b758fbd8000 0 103156 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.554656] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.557289] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.563476] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.568643] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.574405] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.581659] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.586748] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.593021] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.599642] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.605916] kworker/16:0 S ffff8b5965d8e180 0 109801 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.613286] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.615920] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.622110] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.627257] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.633064] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.639805] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.644865] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.651859] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.658677] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.664957] kworker/18:1 S ffff8b6f65eb9040 0 126303 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.672360] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.675014] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.681237] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.686384] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.692156] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.698862] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.704591] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.711044] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.717679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.723948] kworker/42:1 S ffff8b7385bc30c0 0 12111 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.731225] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.733858] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.740048] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.745205] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.750958] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.757667] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.762726] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.769000] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.775643] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.781914] kworker/3:1 S ffff8b8cece74100 0 51272 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.789207] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.791852] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.798044] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.803192] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.808946] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:41 2019][2937563.815676] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:41 2019][2937563.820762] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.827042] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:41 2019][2937563.833668] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:41 2019][2937563.839940] kworker/40:2 S ffff8b5776ffe180 0 55670 2 0x00000080 [Fri Apr 12 11:10:41 2019][2937563.847229] Call Trace: [Fri Apr 12 11:10:41 2019][2937563.849864] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:41 2019][2937563.856052] [] schedule+0x29/0x70 [Fri Apr 12 11:10:41 2019][2937563.861200] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:41 2019][2937563.866956] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937563.873662] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937563.878723] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937563.885007] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937563.891627] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937563.897902] kworker/0:2 S ffff8b5776ffb0c0 0 64678 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937563.905183] Call Trace: [Fri Apr 12 11:10:42 2019][2937563.907820] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937563.914006] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937563.919153] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937563.924946] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937563.931656] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937563.936714] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937563.943011] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937563.949641] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937563.955930] kworker/4:2 S ffff8b5c0f898000 0 66614 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937563.963253] Call Trace: [Fri Apr 12 11:10:42 2019][2937563.965926] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937563.972150] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937563.977311] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937563.983066] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937563.989774] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937563.994840] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.001139] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.007799] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.014076] kworker/1:1 S ffff8b6448e330c0 0 96840 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.021356] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.023994] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937564.030191] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.035336] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.041090] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.048224] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.054256] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:10:42 2019][2937564.060676] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.066954] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.073595] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.079866] kworker/25:1 S ffff8b6448e31040 0 96912 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.087164] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.089800] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.094959] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.100746] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.107484] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.112551] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.118850] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.125494] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.131770] kworker/24:2 S ffff8b5c066e9040 0 102506 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.140234] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.142958] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937564.149159] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.154304] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.160059] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.166766] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.171825] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.178099] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.184761] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.191033] kworker/35:2 S ffff8b8c8dec5140 0 104646 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.198404] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.201036] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937564.207223] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.212372] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.218128] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.224834] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.229895] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.236203] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.242829] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.249118] kworker/26:0 S ffff8b79c36b8000 0 109637 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.256488] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.259124] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.264285] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.270149] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.276871] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.281936] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.288211] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.294828] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.301101] kworker/41:0 S ffff8b5f3476c100 0 113295 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.308464] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.311098] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.316245] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.321998] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.328705] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.333772] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.340091] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.346746] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.353036] kworker/45:0 S ffff8b5e11b4d140 0 120679 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.360430] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.363068] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.368681] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.374479] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.381214] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.386292] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.392602] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.399231] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.405521] kworker/27:2 S ffff8b8ce0834100 0 1805 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.412801] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.415439] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937564.421625] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.426786] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.432536] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.439243] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.444304] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.450580] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.457201] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.463475] kworker/32:1 S ffff8b5c62391040 0 2533 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.470752] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.473396] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937564.479602] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.484761] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.490531] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.497236] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.502297] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.508592] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.515211] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.521486] kworker/2:1 S ffff8b774877b0c0 0 21724 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.528767] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.531413] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.536558] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.542309] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.549022] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.554082] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.560365] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.567018] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.573337] kworker/29:2 S ffff8b69ea63e180 0 22705 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.580620] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.583257] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.588402] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.594154] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.600862] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.605925] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.612202] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.618838] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.625113] kworker/22:0 S ffff8b750a6c6180 0 38528 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.632395] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.635032] [] ? start_worker+0x27/0x30 [Fri Apr 12 11:10:42 2019][2937564.640708] [] ? manage_workers.isra.25+0x189/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.647793] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.653307] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.659067] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.665857] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.670948] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.677241] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.683861] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.690152] kworker/38:0 S ffff8b75d3f11040 0 69869 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.697440] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.700080] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937564.706958] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.712260] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.718015] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.724733] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.729794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.736067] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.742688] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.748961] kworker/34:0 S ffff8b7cf9896180 0 116936 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.756345] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.758985] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:10:42 2019][2937564.765203] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.770357] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.776109] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.782816] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.787924] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.794203] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:42 2019][2937564.800830] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.807100] kworker/u264:2 S ffff8b7cf9891040 0 116989 2 0x00000080 [Fri Apr 12 11:10:42 2019][2937564.814471] Call Trace: [Fri Apr 12 11:10:42 2019][2937564.817106] [] ? process_one_work+0x336/0x440 [Fri Apr 12 11:10:42 2019][2937564.823314] [] schedule+0x29/0x70 [Fri Apr 12 11:10:42 2019][2937564.828468] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:10:42 2019][2937564.834241] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:10:42 2019][2937564.840965] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:42 2019][2937564.846028] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:42 2019][2937564.852299] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937564.858917] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937564.865184] cfs_rh_00 S ffff8b8cf1ef0000 0 117052 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937564.872558] Call Trace: [Fri Apr 12 11:10:43 2019][2937564.875188] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937564.880793] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Fri Apr 12 11:10:43 2019][2937564.887596] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:43 2019][2937564.893615] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Fri Apr 12 11:10:43 2019][2937564.900845] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937564.905948] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937564.912233] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937564.918881] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937564.925164] cfs_rh_01 S ffff8b8cf1ef1040 0 117053 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937564.932523] Call Trace: [Fri Apr 12 11:10:43 2019][2937564.935158] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937564.940319] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Fri Apr 12 11:10:43 2019][2937564.947128] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:43 2019][2937564.953149] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Fri Apr 12 11:10:43 2019][2937564.960377] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937564.965454] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937564.971730] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937564.978348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937564.984670] cfs_rh_02 S ffff8b8cf1ef30c0 0 117054 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937564.992060] Call Trace: [Fri Apr 12 11:10:43 2019][2937564.994902] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.000555] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Fri Apr 12 11:10:43 2019][2937565.007526] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:43 2019][2937565.013567] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Fri Apr 12 11:10:43 2019][2937565.020805] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.025882] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.032157] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.038786] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.045493] cfs_rh_03 S ffff8b8cf1ef4100 0 117055 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.053418] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.056063] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.061221] [] cfs_wi_scheduler+0x30c/0x460 [libcfs] [Fri Apr 12 11:10:43 2019][2937565.068036] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:43 2019][2937565.074062] [] ? cfs_wi_sched_create+0x680/0x680 [libcfs] [Fri Apr 12 11:10:43 2019][2937565.081288] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.086375] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.092651] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.099274] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.106147] clio S ffff8b8cfa5f1040 0 117068 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.113886] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.116520] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.121676] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:43 2019][2937565.127528] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:43 2019][2937565.133716] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.138784] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.145059] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.152204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.158919] ptlrpc_hr00_000 S ffff8b8cfa5f4100 0 117073 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.166307] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.168948] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.175055] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.180254] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.186876] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.192711] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.199719] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.204782] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.211119] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.217766] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.224038] ptlrpc_hr00_001 S ffff8b8cfa5f2080 0 117074 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.231792] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.234970] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.241220] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.250254] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.257140] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.262947] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.269932] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.275008] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.281287] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.287915] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.294189] ptlrpc_hr00_002 S ffff8b8cfa5f5140 0 117075 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.301584] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.304215] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.310716] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.316819] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.323461] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.329276] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.336266] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.341329] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.347621] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.354276] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.360548] ptlrpc_hr00_003 S ffff8b8cfa5f30c0 0 117076 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.367913] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.370543] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.376645] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.381837] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.388460] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.394253] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.401222] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.406295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.412581] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.419219] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.425537] ptlrpc_hr00_004 S ffff8b8cfa5f6180 0 117077 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.432906] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.435548] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.441664] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.446878] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.453507] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.459340] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.466316] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.471392] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.477667] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.484287] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.490559] ptlrpc_hr00_005 S ffff8b7da94e30c0 0 117078 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.497933] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.500567] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.506676] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.511872] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.518501] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.524307] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.531281] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.536342] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.542620] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.549256] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.555537] ptlrpc_hr01_000 S ffff8b7da94e2080 0 117080 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.562899] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.565533] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.571652] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.576853] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.583513] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.589361] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.596343] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.601405] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.607710] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.614358] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.620633] ptlrpc_hr01_001 S ffff8b7da94e0000 0 117081 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.628016] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.630675] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.636800] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.642016] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.648651] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.654582] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.662013] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.667485] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.673814] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.680448] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.686737] ptlrpc_hr01_002 S ffff8b8cf6b66180 0 117082 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.694161] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.696801] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.702901] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.708133] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.714770] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.720564] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.727551] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.732613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.738897] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.745518] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.751791] ptlrpc_hr01_003 S ffff8b8cf6b60000 0 117083 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.759203] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.761840] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.767956] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.773150] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.779777] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.785582] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.792569] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.797648] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.803924] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:43 2019][2937565.810569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.816872] ptlrpc_hr01_004 S ffff8b895eefe180 0 117084 2 0x00000080 [Fri Apr 12 11:10:43 2019][2937565.824259] Call Trace: [Fri Apr 12 11:10:43 2019][2937565.826901] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:43 2019][2937565.833006] [] schedule+0x29/0x70 [Fri Apr 12 11:10:43 2019][2937565.838206] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.844829] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:43 2019][2937565.850626] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:43 2019][2937565.857615] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:43 2019][2937565.862673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:43 2019][2937565.869019] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937565.875644] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937565.881912] ptlrpc_hr01_005 S ffff8b895eefd140 0 117085 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937565.889295] Call Trace: [Fri Apr 12 11:10:44 2019][2937565.891929] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937565.898040] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937565.903265] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937565.909924] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937565.915730] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937565.922722] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937565.927816] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937565.934090] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937565.940775] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937565.947501] ptlrpc_hr02_000 S ffff8b895eefc100 0 117086 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937565.955422] Call Trace: [Fri Apr 12 11:10:44 2019][2937565.958057] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937565.964620] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937565.970050] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937565.976708] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937565.982520] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937565.989526] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937565.994589] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.000878] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.007516] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.013789] ptlrpc_hr02_001 S ffff8b895eef8000 0 117087 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.021188] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.023829] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.029927] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.035126] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.041753] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.047557] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.054555] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.059656] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.065940] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.072567] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.078885] ptlrpc_hr02_002 S ffff8b895eefa080 0 117088 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.086293] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.088935] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.095044] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.100269] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.106911] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.112730] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.119704] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.124788] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.131074] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.137694] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.143964] ptlrpc_hr02_003 S ffff8b895eefb0c0 0 117089 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.151325] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.153959] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.160060] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.165257] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.171873] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.177669] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.184638] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.189699] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.195973] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.202595] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.208868] ptlrpc_hr02_004 S ffff8b8ccbe330c0 0 117090 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.216223] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.218855] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.224956] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.230149] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.236770] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.242564] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.249528] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.254593] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.260894] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.267569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.273863] ptlrpc_hr02_005 S ffff8b8ccbe35140 0 117091 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.281227] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.283876] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.289977] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.295172] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.301808] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.307616] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.314595] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.319679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.325977] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.332604] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.338885] ptlrpc_hr03_000 S ffff8b8ccbe30000 0 117093 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.346285] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.348948] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.355060] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.364231] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.371159] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.377029] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.384033] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.389143] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.395441] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.402080] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.408423] ptlrpc_hr03_001 S ffff8b8ccbe34100 0 117094 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.416667] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.419320] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.425424] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.430622] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.437268] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.443088] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.450064] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.455133] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.461435] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.468575] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.474859] ptlrpc_hr03_002 S ffff8b7da9725140 0 117095 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.482235] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.484871] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.490970] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.496201] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.503225] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.509035] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.516014] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.521080] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.527354] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.533973] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.540277] ptlrpc_hr03_003 S ffff8b7da9724100 0 117096 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.547663] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.550297] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.556396] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.561594] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.568222] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.574050] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.581038] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.586096] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.592373] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.599001] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.605305] ptlrpc_hr03_004 S ffff8b7da9722080 0 117097 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.612701] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.615363] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.621467] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.626662] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.633280] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.639073] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.646037] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.651098] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.657374] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.663992] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.670267] ptlrpc_hr03_005 S ffff8b7da97230c0 0 117098 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.677631] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.680262] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:44 2019][2937566.686364] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.691554] [] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.698812] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.704736] [] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc] [Fri Apr 12 11:10:44 2019][2937566.711708] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.716769] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.723042] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.729663] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.735936] kiblnd_connd S ffff8b7da9726180 0 117113 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.743985] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.746753] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:10:44 2019][2937566.752173] [] schedule+0x29/0x70 [Fri Apr 12 11:10:44 2019][2937566.757321] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:44 2019][2937566.763348] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:44 2019][2937566.769907] [] kiblnd_connd+0x60c/0xa00 [ko2iblnd] [Fri Apr 12 11:10:44 2019][2937566.776529] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:44 2019][2937566.782306] [] ? kiblnd_check_conns+0x870/0x870 [ko2iblnd] [Fri Apr 12 11:10:44 2019][2937566.789651] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:44 2019][2937566.794718] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.801003] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:44 2019][2937566.807634] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:44 2019][2937566.813924] kiblnd_sd_00_00 R running task 0 117115 2 0x00000080 [Fri Apr 12 11:10:44 2019][2937566.821936] Call Trace: [Fri Apr 12 11:10:44 2019][2937566.824595] [] ? __enqueue_entity+0x78/0x80 [Fri Apr 12 11:10:44 2019][2937566.830771] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:10:44 2019][2937566.836633] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:10:44 2019][2937566.843271] [] ? set_next_entity+0x3c/0xe0 [Fri Apr 12 11:10:44 2019][2937566.849206] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:10:44 2019][2937566.855180] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:10:44 2019][2937566.860947] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:10:45 2019][2937566.868106] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:45 2019][2937566.874653] [] ? schedule+0x29/0x70 [Fri Apr 12 11:10:45 2019][2937566.880011] [] ? kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937566.887238] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937566.893520] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937566.899652] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937566.905761] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937566.912659] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937566.917891] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937566.924185] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937566.931376] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937566.937987] kiblnd_sd_00_01 S ffff8b8584a21040 0 117116 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937566.945370] Call Trace: [Fri Apr 12 11:10:45 2019][2937566.948015] [] ? kiblnd_pool_free_node+0x82/0x170 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937566.955528] [] ? schedule+0x29/0x70 [Fri Apr 12 11:10:45 2019][2937566.960888] [] ? kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937566.968145] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937566.974418] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937566.980085] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937566.985884] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937566.992776] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937566.998008] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.004332] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.011139] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.017413] kiblnd_sd_00_02 S 0000000000000001 0 117117 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.024784] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.027416] [] schedule+0x29/0x70 [Fri Apr 12 11:10:45 2019][2937567.032570] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.039631] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.045921] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937567.051584] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937567.057346] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.064221] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937567.069280] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.075552] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.082185] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.088462] kiblnd_sd_00_03 R running task 0 117118 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.095835] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.098477] [] ? native_queued_spin_lock_slowpath+0x1d6/0x200 [Fri Apr 12 11:10:45 2019][2937567.106102] [] __cond_resched+0x26/0x30 [Fri Apr 12 11:10:45 2019][2937567.111770] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.118057] [] ? lnet_parse+0x4ae/0xff0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.124329] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:10:45 2019][2937567.130967] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:10:45 2019][2937567.137193] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:10:45 2019][2937567.142609] [] ? schedule+0x29/0x70 [Fri Apr 12 11:10:45 2019][2937567.147938] [] ? kiblnd_scheduler+0x89d/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.155174] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.161457] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937567.167133] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937567.172904] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.179787] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937567.185031] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.191316] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.198118] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.204391] kiblnd_sd_01_00 R running task 0 117119 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.211760] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.214410] [] ? lnet_parse_local+0x5ae/0xd40 [lnet] [Fri Apr 12 11:10:45 2019][2937567.221218] [] __cond_resched+0x26/0x30 [Fri Apr 12 11:10:45 2019][2937567.227381] [] _cond_resched+0x3a/0x50 [Fri Apr 12 11:10:45 2019][2937567.233297] [] ? lnet_parse+0x8ea/0xff0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.240403] [] ? mlx5_ib_poll_cq+0x4c5/0xf10 [mlx5_ib] [Fri Apr 12 11:10:45 2019][2937567.247758] [] ? kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.254929] [] ? kiblnd_scheduler+0xf3c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.262166] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.268951] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937567.274667] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937567.280437] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.287326] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937567.292560] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.298836] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.305631] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.311912] kiblnd_sd_01_01 R running task 0 117120 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.319283] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.321930] [] ? lnet_recv_put+0x85/0xb0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.329496] [] ? lnet_parse_local+0x5ae/0xd40 [lnet] [Fri Apr 12 11:10:45 2019][2937567.336982] [] ? __cond_resched+0x26/0x30 [Fri Apr 12 11:10:45 2019][2937567.342819] [] ? _cond_resched+0x3a/0x50 [Fri Apr 12 11:10:45 2019][2937567.348578] [] ? kmem_cache_alloc_trace+0x3c/0x200 [Fri Apr 12 11:10:45 2019][2937567.355209] [] ? lnet_parse+0x4ae/0xff0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.361497] [] ? lnet_parse+0x8ea/0xff0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.367787] [] ? kiblnd_check_sends_locked+0x92e/0xe40 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.375725] [] ? kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.382887] [] ? kiblnd_scheduler+0xf3c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.390117] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.396390] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937567.402058] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937567.407832] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.414712] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937567.419946] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.426231] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.433602] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.440094] kiblnd_sd_01_02 R running task 0 117121 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.447474] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.450109] [] schedule+0x29/0x70 [Fri Apr 12 11:10:45 2019][2937567.455279] [] ? kiblnd_scheduler+0xf3c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.462538] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.468811] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937567.474496] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937567.480258] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.487157] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937567.492393] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.498669] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.505479] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.511753] kiblnd_sd_01_03 R running task 0 117122 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.519125] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.521815] [] ? lnet_recv_put+0x85/0xb0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.528190] [] ? kfree+0x106/0x140 [Fri Apr 12 11:10:45 2019][2937567.533441] [] ? lnet_parse_local+0x569/0xd40 [lnet] [Fri Apr 12 11:10:45 2019][2937567.540261] [] ? __cond_resched+0x26/0x30 [Fri Apr 12 11:10:45 2019][2937567.546102] [] ? __radix_tree_lookup+0x84/0xf0 [Fri Apr 12 11:10:45 2019][2937567.552383] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:45 2019][2937567.559273] [] ? lnet_parse+0x4ae/0xff0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.565551] [] ? kiblnd_check_sends_locked+0x92e/0xe40 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.573498] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [Fri Apr 12 11:10:45 2019][2937567.581390] [] ? kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.589046] [] ? kiblnd_scheduler+0xf3c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.596285] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.602897] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937567.608643] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937567.614410] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.621331] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937567.626566] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.632903] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.640341] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.646750] kiblnd_sd_02_00 R running task 0 117123 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.654120] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.656762] [] schedule+0x29/0x70 [Fri Apr 12 11:10:45 2019][2937567.661921] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.669002] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.675911] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937567.681749] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937567.687535] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.694479] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937567.699567] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.705849] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.712472] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.718741] kiblnd_sd_02_01 R running task 0 117124 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.726106] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.729316] [] ? list_del+0xd/0x30 [Fri Apr 12 11:10:45 2019][2937567.734946] [] schedule+0x29/0x70 [Fri Apr 12 11:10:45 2019][2937567.740123] [] ? kiblnd_scheduler+0xf3c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.747356] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.753649] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:45 2019][2937567.759313] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:45 2019][2937567.765075] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:45 2019][2937567.771961] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:45 2019][2937567.777212] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.783489] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:45 2019][2937567.790295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:45 2019][2937567.797143] kiblnd_sd_02_02 R running task 0 117125 2 0x00000080 [Fri Apr 12 11:10:45 2019][2937567.804862] Call Trace: [Fri Apr 12 11:10:45 2019][2937567.807521] [] ? lnet_recv_put+0x85/0xb0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.813889] [] ? lnet_parse_local+0x5ae/0xd40 [lnet] [Fri Apr 12 11:10:45 2019][2937567.820684] [] ? __cond_resched+0x26/0x30 [Fri Apr 12 11:10:45 2019][2937567.826525] [] ? ttwu_do_wakeup+0xb5/0xe0 [Fri Apr 12 11:10:45 2019][2937567.832365] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:45 2019][2937567.838643] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:10:45 2019][2937567.844600] [] ? lnet_parse+0x8ea/0xff0 [lnet] [Fri Apr 12 11:10:45 2019][2937567.850880] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:10:46 2019][2937567.857522] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:10:46 2019][2937567.863720] [] ? schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937567.869069] [] ? kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937567.876854] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:46 2019][2937567.883429] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:46 2019][2937567.889114] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937567.894888] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937567.901782] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937567.907017] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937567.913301] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937567.920094] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937567.926375] kiblnd_sd_02_03 S ffff8b8cc81b3c78 0 117126 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937567.933738] Call Trace: [Fri Apr 12 11:10:46 2019][2937567.936369] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937567.941524] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937567.948581] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:46 2019][2937567.954855] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:46 2019][2937567.960522] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937567.966294] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937567.973660] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937567.978795] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937567.985068] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937567.991689] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937567.997965] kiblnd_sd_03_00 S ffff8b8cf92a2080 0 117127 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.005338] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.007979] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.013140] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937568.020197] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:46 2019][2937568.026472] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:46 2019][2937568.032166] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.037923] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937568.044810] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.049887] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.056202] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.062829] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.069117] kiblnd_sd_03_01 R running task 0 117128 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.076479] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.079114] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.084264] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937568.091317] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:46 2019][2937568.097635] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:46 2019][2937568.103856] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.110076] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937568.116984] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.122069] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.128354] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.135005] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.141279] kiblnd_sd_03_02 S ffff8b8ce0835140 0 117129 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.148668] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.151301] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.156464] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937568.164205] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:46 2019][2937568.170685] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:46 2019][2937568.176377] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.182172] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937568.189106] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.194173] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.200447] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.207068] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.213345] kiblnd_sd_03_03 R running task 0 117130 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.220706] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.223351] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:46 2019][2937568.229835] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.234996] [] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937568.242076] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:46 2019][2937568.248352] [] ? __switch_to+0xce/0x580 [Fri Apr 12 11:10:46 2019][2937568.254061] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.259825] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [Fri Apr 12 11:10:46 2019][2937568.266705] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.271765] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.278041] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.284663] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.290935] monitor_thread S ffff8b8ce0831040 0 117131 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.298366] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.301029] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.306194] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:46 2019][2937568.312213] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:46 2019][2937568.318754] [] ? prepare_to_wait+0x56/0x90 [Fri Apr 12 11:10:46 2019][2937568.324692] [] lnet_monitor_thread+0x492/0x560 [lnet] [Fri Apr 12 11:10:46 2019][2937568.331577] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:46 2019][2937568.337603] [] ? lnet_recover_peer_nis+0x670/0x670 [lnet] [Fri Apr 12 11:10:46 2019][2937568.344829] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.349891] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.356165] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.362790] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.369061] lnet_discovery S ffff8b8ce08330c0 0 117132 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.376468] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.379098] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.384247] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:46 2019][2937568.390268] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:46 2019][2937568.396806] [] ? prepare_to_wait+0x56/0x90 [Fri Apr 12 11:10:46 2019][2937568.402770] [] lnet_peer_discovery+0xdf/0x1150 [lnet] [Fri Apr 12 11:10:46 2019][2937568.409661] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:46 2019][2937568.415702] [] ? lnet_peer_merge_data+0xde0/0xde0 [lnet] [Fri Apr 12 11:10:46 2019][2937568.422866] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.427929] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.434902] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.441644] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.447934] ptlrpcd_rcv S ffff8b8ce0830000 0 117133 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.455309] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.457978] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:46 2019][2937568.464714] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.469908] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.475947] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.481767] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.488485] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.493549] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.499821] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.507006] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.513365] ptlrpcd_00_00 S ffff8b8ce0832080 0 117134 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.520764] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.523439] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:46 2019][2937568.530408] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.535802] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.541822] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.547618] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.554339] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.559921] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.566298] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.573408] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.579822] ptlrpcd_00_01 S ffff8b7da95b8000 0 117135 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.587184] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.589857] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:46 2019][2937568.596571] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.601768] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.607802] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.613595] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.620304] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.625368] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.631643] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.638273] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.644553] ptlrpcd_00_02 S ffff8b7da95bb0c0 0 117136 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.651934] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.654600] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:46 2019][2937568.661313] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.666501] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.672515] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.678319] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.685032] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.690110] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.696469] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.703127] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.709409] ptlrpcd_00_03 S ffff8b8cf31fe180 0 117137 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.716779] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.719482] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:46 2019][2937568.726205] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.731402] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.737425] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.743218] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.750397] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.755456] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.761759] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.768399] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.774681] ptlrpcd_00_04 S ffff8b8cf31fc100 0 117138 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.782095] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.785049] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:46 2019][2937568.791785] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.796998] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.803035] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:46 2019][2937568.808854] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:46 2019][2937568.815562] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:46 2019][2937568.820671] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.828341] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:46 2019][2937568.835067] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:46 2019][2937568.841334] ptlrpcd_00_05 S ffff8b8cf31fd140 0 117139 2 0x00000080 [Fri Apr 12 11:10:46 2019][2937568.848699] Call Trace: [Fri Apr 12 11:10:46 2019][2937568.851363] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:46 2019][2937568.858072] [] schedule+0x29/0x70 [Fri Apr 12 11:10:46 2019][2937568.863269] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937568.869278] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937568.875072] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937568.881776] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937568.886845] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937568.893125] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937568.899743] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937568.906026] ptlrpcd_00_06 S ffff8b8cf31f8000 0 117140 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937568.913388] Call Trace: [Fri Apr 12 11:10:47 2019][2937568.916055] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937568.922764] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937568.927965] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937568.933979] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937568.939774] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937568.946486] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937568.951572] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937568.957859] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937568.965027] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937568.971375] ptlrpcd_00_07 S ffff8b8cf31fb0c0 0 117141 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937568.978735] Call Trace: [Fri Apr 12 11:10:47 2019][2937568.981404] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937568.988110] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937568.993304] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937568.999318] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.005110] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.011815] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.016873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.023163] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.029780] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.036053] ptlrpcd_00_08 S ffff8b8cf31f9040 0 117142 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.043416] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.046082] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.052793] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.057986] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.063996] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.069791] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.076565] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.081634] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.087920] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.094540] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.100812] ptlrpcd_00_09 S ffff8b8c5fd5a080 0 117143 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.108178] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.110851] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.117577] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.127008] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.133283] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.139104] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.146350] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.151769] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.158160] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.164785] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.171055] ptlrpcd_00_10 S ffff8b8c5fd5b0c0 0 117144 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.178412] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.181075] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.187787] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.192981] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.199022] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.204822] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.211546] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.216617] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.222898] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.229517] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.235790] ptlrpcd_00_11 S ffff8b8c5fd5e180 0 117145 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.243152] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.245822] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.252527] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.257722] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.263732] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.269537] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.276286] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.281725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.288380] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.295034] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.301339] ptlrpcd_01_00 S ffff8b8c5fd5c100 0 117146 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.308729] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.311415] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.318137] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.323343] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.329380] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.335216] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.341946] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.347016] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.353308] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.359926] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.366201] ptlrpcd_01_01 S ffff8b8c5fd58000 0 117147 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.373574] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.376248] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.382956] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.388160] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.394183] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.400301] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.407050] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.412108] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.418386] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.425006] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.431279] ptlrpcd_01_02 S ffff8b8c5fd59040 0 117148 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.438643] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.441305] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.448037] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.453234] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.459250] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.465042] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.471761] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.476828] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.483193] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.489821] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.496101] ptlrpcd_01_03 S ffff8b8cf8b20000 0 117149 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.503462] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.506129] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.512837] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.518031] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.524042] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.529838] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.536550] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.541610] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.547893] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.554514] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.560788] ptlrpcd_01_04 S ffff8b8cf8b25140 0 117150 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.568153] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.570812] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.577517] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.582707] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.588725] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.594523] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.601230] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.606291] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.612567] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.619185] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.625460] ptlrpcd_01_05 S ffff8b8cf8b230c0 0 117151 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.632823] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.635484] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.642188] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.647381] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.653396] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.659193] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.665901] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.670961] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.677236] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.683858] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.690132] ptlrpcd_01_06 S ffff8b8cf8b24100 0 117152 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.697488] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.700150] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.706874] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.712077] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.718114] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.723907] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.730636] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.735724] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.742020] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.748664] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.754952] ptlrpcd_01_07 S ffff8b8577f030c0 0 117153 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.762325] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.764994] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.771710] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.776915] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.782931] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.788725] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.795438] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:47 2019][2937569.800498] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.806774] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:47 2019][2937569.813396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:47 2019][2937569.819667] ptlrpcd_01_08 S ffff8b8577f02080 0 117154 2 0x00000080 [Fri Apr 12 11:10:47 2019][2937569.827031] Call Trace: [Fri Apr 12 11:10:47 2019][2937569.829695] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:47 2019][2937569.836451] [] schedule+0x29/0x70 [Fri Apr 12 11:10:47 2019][2937569.841642] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:47 2019][2937569.847685] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:47 2019][2937569.853495] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937569.860204] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937569.865848] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937569.872434] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937569.879074] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937569.885372] ptlrpcd_01_09 S ffff8b8577f01040 0 117155 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937569.892735] Call Trace: [Fri Apr 12 11:10:48 2019][2937569.895405] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937569.902139] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937569.907341] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937569.913370] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937569.919168] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937569.925875] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937569.930937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937569.937222] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937569.943852] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937569.950512] ptlrpcd_01_10 S ffff8b8577f04100 0 117156 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937569.957998] Call Trace: [Fri Apr 12 11:10:48 2019][2937569.960790] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937569.967507] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937569.972710] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937569.978723] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937569.984513] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937569.991241] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937569.996581] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.002898] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.009545] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.015816] ptlrpcd_01_11 S ffff8b8577f06180 0 117157 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.023180] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.025846] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.032554] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.037745] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.043760] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.049583] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.056293] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.061390] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.068118] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.074818] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.081105] ptlrpcd_02_00 S ffff8b8577f05140 0 117158 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.088468] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.091132] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.097841] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.103038] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.109065] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.114861] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.121569] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.126634] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.132924] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.139545] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.145819] ptlrpcd_02_01 S ffff8b8cfc819040 0 117159 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.153181] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.155843] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.162548] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.167740] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.173757] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.179548] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.186251] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.191324] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.197670] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.204317] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.210586] ptlrpcd_02_02 S ffff8b8cfc81d140 0 117160 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.217970] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.220649] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.227359] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.232552] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.238585] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.244408] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.251144] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.256219] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.262504] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.269153] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.275434] ptlrpcd_02_03 S ffff8b8cfc81c100 0 117161 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.282804] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.285470] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.292197] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.297407] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.303419] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.309214] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.315946] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.321002] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.328007] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.334851] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.341172] ptlrpcd_02_04 S ffff8b8cfc818000 0 117162 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.348561] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.351226] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.357933] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.363130] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.369142] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.374932] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.381657] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.386743] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.393340] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.399983] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.406258] ptlrpcd_02_05 S ffff8b8cfc81e180 0 117163 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.413745] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.416431] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.423162] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.431132] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.437191] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.443047] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.449816] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.454924] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.461219] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.467858] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.474130] ptlrpcd_02_06 S ffff8b86da3cd140 0 117164 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.481532] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.484205] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.490947] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.496142] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.502163] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.507967] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.514694] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.519753] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.526048] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.532671] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.538978] ptlrpcd_02_07 S ffff8b86da3cb0c0 0 117165 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.546340] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.549032] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.555751] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.560942] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.566966] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.572768] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.579480] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.584538] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.590817] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.597445] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.603734] ptlrpcd_02_08 S ffff8b86da3c8000 0 117166 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.611113] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.613782] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.620507] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.625699] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.631723] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.637545] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.644802] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.649864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.656154] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.662819] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.669122] ptlrpcd_02_09 S ffff8b86da3c9040 0 117167 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.676483] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.679156] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.685862] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.691057] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.697082] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.702902] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.709628] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.714688] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.720966] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.727652] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.733934] ptlrpcd_02_10 S ffff8b86da3cc100 0 117168 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.741290] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.743956] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.750667] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.755857] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.761871] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.767736] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.774463] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.779560] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.785835] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:48 2019][2937570.792474] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.798748] ptlrpcd_02_11 S ffff8b86da3ca080 0 117169 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.806129] Call Trace: [Fri Apr 12 11:10:48 2019][2937570.808803] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:48 2019][2937570.815521] [] schedule+0x29/0x70 [Fri Apr 12 11:10:48 2019][2937570.820711] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.826726] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:48 2019][2937570.832514] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:48 2019][2937570.839224] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:48 2019][2937570.844707] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.851249] [] re[2937570.857876] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:48 2019][2937570.864193] ptlrpcd_03_00 S ffff8b8886324100 0 117170 2 0x00000080 [Fri Apr 12 11:10:48 2019][2937570.871582] Call Trace: [Fri Apr 12 11:10:49 2019][2937570.874271] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937570.882142] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937570.887289] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:49 2019][2937570.893303] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:49 2019][2937570.899883] [] ptlrpcd+0x278/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937570.905895] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937570.911694] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937570.918403] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937570.923465] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937570.929739] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937570.936357] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937570.942633] ptlrpcd_03_01 S ffff8b8886320000 0 117171 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937570.949996] Call Trace: [Fri Apr 12 11:10:49 2019][2937570.952658] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:49 2019][2937570.959362] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937570.964552] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937570.970567] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937570.976359] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937570.983065] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937570.988126] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937570.994402] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.001023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.007295] ptlrpcd_03_02 S ffff8b8886321040 0 117172 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.014651] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.017325] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.025162] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.030308] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:49 2019][2937571.036323] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:49 2019][2937571.042901] [] ptlrpcd+0x278/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.048915] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.054711] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.061422] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.066485] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.072761] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.079384] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.085673] ptlrpcd_03_03 S ffff8b8886322080 0 117173 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.093035] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.095700] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:49 2019][2937571.102430] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.107628] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.113654] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.119465] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.126183] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.131245] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.137518] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.144139] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.150412] ptlrpcd_03_04 S ffff8b8886325140 0 117174 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.157787] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.160468] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:49 2019][2937571.167177] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.172406] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.178723] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.185194] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.191922] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.196982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.203279] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.209904] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.216187] ptlrpcd_03_05 S ffff8b8886326180 0 117175 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.223549] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.226230] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.234080] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.239225] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:49 2019][2937571.245242] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:49 2019][2937571.251820] [] ptlrpcd+0x278/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.257837] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.263639] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.270352] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.275417] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.281713] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.288346] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.294626] ptlrpcd_03_06 S ffff8b8cec68a080 0 117176 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.301993] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.304673] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.312513] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.317660] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:49 2019][2937571.323675] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:49 2019][2937571.330250] [] ptlrpcd+0x278/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.336266] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.342059] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.348763] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.353824] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.360101] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.366722] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.372993] ptlrpcd_03_07 S ffff8b8cec689040 0 117177 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.380351] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.383013] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:49 2019][2937571.389724] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.394913] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.400931] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.406724] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.413425] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.418488] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.424764] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.431383] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.437648] ptlrpcd_03_08 S ffff8b8cec68c100 0 117178 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.445004] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.447663] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:49 2019][2937571.454368] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.459559] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.465576] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.471369] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.478072] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.483132] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.489400] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.496022] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.502311] ptlrpcd_03_09 S ffff8b8cec688000 0 117179 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.509669] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.512339] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.520177] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.525326] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:49 2019][2937571.531341] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:49 2019][2937571.537916] [] ptlrpcd+0x278/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.543932] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.549727] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.556431] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.561493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.567768] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.574389] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.580660] ptlrpcd_03_10 S ffff8b8cec68e180 0 117180 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.588017] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.590679] [] ? lu_env_refill+0x24/0x30 [obdclass] [Fri Apr 12 11:10:49 2019][2937571.597391] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.602581] [] ptlrpcd+0x2ee/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.608595] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.614393] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.621103] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.626164] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.632438] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.639059] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.645332] ptlrpcd_03_11 S ffff8b8cec68d140 0 117181 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.652687] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.655360] [] ? ptlrpc_check_set.part.23+0x11bd/0x1df0 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.663198] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.668345] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:49 2019][2937571.674361] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:49 2019][2937571.680936] [] ptlrpcd+0x278/0x550 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.686954] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:49 2019][2937571.692747] [] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [Fri Apr 12 11:10:49 2019][2937571.699450] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.704512] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.710785] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.717409] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.723682] kmmpd-dm-1 S ffff8b87e2339040 0 117246 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.731045] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.733677] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.738823] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:49 2019][2937571.744856] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:49 2019][2937571.751401] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Fri Apr 12 11:10:49 2019][2937571.758367] [] schedule_timeout_interruptible+0x1e/0x20 [Fri Apr 12 11:10:49 2019][2937571.765432] [] kmmpd+0x216/0x430 [ldiskfs] [Fri Apr 12 11:10:49 2019][2937571.771368] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Fri Apr 12 11:10:49 2019][2937571.778075] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.783138] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.789414] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.796035] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.802307] jbd2/dm-1-8 S ffff8b87e2338000 0 117247 2 0x00000080 [Fri Apr 12 11:10:49 2019][2937571.809672] Call Trace: [Fri Apr 12 11:10:49 2019][2937571.812302] [] schedule+0x29/0x70 [Fri Apr 12 11:10:49 2019][2937571.817467] [] kjournald2+0x203/0x260 [jbd2] [Fri Apr 12 11:10:49 2019][2937571.823569] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:49 2019][2937571.829588] [] ? commit_timeout+0x10/0x10 [jbd2] [Fri Apr 12 11:10:49 2019][2937571.836030] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:49 2019][2937571.841093] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:49 2019][2937571.847369] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:49 2019][2937571.853988] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937571.860263] ldiskfs-rsv-con S ffff8b87e233a080 0 117248 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937571.867634] Call Trace: [Fri Apr 12 11:10:50 2019][2937571.870267] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937571.875413] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:50 2019][2937571.881256] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:50 2019][2937571.887441] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937571.892496] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937571.898773] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937571.905392] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937571.911664] ldlm_cb00_000 S ffff8b87e233c100 0 117256 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937571.919022] Call Trace: [Fri Apr 12 11:10:50 2019][2937571.921652] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937571.926849] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937571.933725] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937571.939528] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937571.946011] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937571.953579] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937571.958640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937571.964915] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937571.971537] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937571.977811] lc_watchdogd S ffff8b87e233d140 0 117257 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937571.985166] Call Trace: [Fri Apr 12 11:10:50 2019][2937571.987800] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937571.992952] [] lcw_dispatch_main+0x749/0x940 [libcfs] [Fri Apr 12 11:10:50 2019][2937571.999829] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:50 2019][2937572.005850] [] ? lc_watchdog_add+0x790/0x790 [libcfs] [Fri Apr 12 11:10:50 2019][2937572.012734] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.017794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.024072] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.030719] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.036999] ldlm_cb00_001 S ffff8b87e233e180 0 117258 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.044363] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.046997] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.052201] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.059085] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.064890] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.071386] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.078966] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.084024] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.090300] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.096939] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.103213] ldlm_cb01_000 S ffff8b8b872c9040 0 117259 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.110577] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.113210] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.118399] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.125282] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.131076] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.141484] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.149089] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.154152] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.160438] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.167093] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.173383] ldlm_cb01_001 S ffff8b8b872c8000 0 117260 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.180751] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.183396] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.188585] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.195481] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.201317] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.207831] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.215444] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.220528] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.226845] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.233469] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.239760] ldlm_cb02_000 S ffff8b8b872cb0c0 0 117261 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.247123] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.249754] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.254950] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.261829] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.267623] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.274113] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.281698] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.286760] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.293036] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.299659] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.305930] ldlm_cb02_001 S ffff8b8b872cc100 0 117263 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.313298] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.315949] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.321158] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.328047] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.333859] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.340346] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.347953] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.353018] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.359330] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.365962] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.372231] ldlm_cb03_000 S ffff8b8b872ca080 0 117264 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.379588] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.382221] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.387414] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.394293] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.400082] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.406567] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.414142] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.419200] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.425497] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.432112] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.438385] ldlm_cb03_001 S ffff8b8b872ce180 0 117265 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.445742] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.448371] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.453565] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.460455] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.466250] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.472737] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.480307] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.485365] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.491641] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.498278] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.504551] ldlm_cn00_000 S ffff8b8b38a24100 0 117266 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.511935] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.514574] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.519764] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.526654] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.532454] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.538935] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.546510] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.551586] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.557864] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.564499] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.570774] ldlm_cn00_001 S ffff8b8b38a21040 0 117267 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.578154] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.580800] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.586016] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.592897] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.598698] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.605213] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.612830] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.617888] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.624162] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.630783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.637056] ldlm_cn00_002 S ffff8b8b38a230c0 0 117268 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.644412] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.647046] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.652236] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.659156] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.664972] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.671463] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.679073] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.684139] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.690413] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.697033] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.703307] ldlm_cn01_000 S ffff8b8b38a20000 0 117269 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.710670] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.713302] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.718497] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.725393] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.731189] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.737676] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.745315] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.750389] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.756673] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.763293] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.769563] ldlm_cn01_001 S ffff8b8b38a22080 0 117270 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.776957] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.779596] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.784802] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.791698] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:50 2019][2937572.797588] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.804144] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.811751] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:50 2019][2937572.816821] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.823115] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:50 2019][2937572.829786] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:50 2019][2937572.836078] ldlm_cn01_002 S ffff8b8b38a26180 0 117271 2 0x00000080 [Fri Apr 12 11:10:50 2019][2937572.843474] Call Trace: [Fri Apr 12 11:10:50 2019][2937572.846107] [] schedule+0x29/0x70 [Fri Apr 12 11:10:50 2019][2937572.851297] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:50 2019][2937572.858182] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937572.864025] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937572.870526] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937572.878129] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937572.883198] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937572.889508] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937572.896132] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937572.902403] ldlm_cn02_000 S ffff8b8bc622a080 0 117272 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937572.909783] Call Trace: [Fri Apr 12 11:10:51 2019][2937572.912418] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937572.917618] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937572.924510] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937572.930325] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937572.936814] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937572.944412] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937572.949476] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937572.955771] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937572.963007] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937572.969436] ldlm_cn02_001 S ffff8b8bc6228000 0 117273 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937572.976824] Call Trace: [Fri Apr 12 11:10:51 2019][2937572.979477] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937572.984694] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937572.991609] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937572.997438] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.003948] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.011540] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.016619] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.022896] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.029521] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.035792] ldlm_cn02_002 S ffff8b8bc622d140 0 117274 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.043149] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.045780] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.050975] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.057853] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.063649] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.070138] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.077707] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.082768] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.089044] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.095665] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.101941] ldlm_cn03_000 S ffff8b8bc6229040 0 117275 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.109312] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.111946] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.117154] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.124038] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.133955] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.140607] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.148216] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.153288] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.159570] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.166207] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.172496] ldlm_cn03_001 S ffff8b8bc622c100 0 117276 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.179877] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.182516] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.187737] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.194620] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.204797] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.211558] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.219140] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.224197] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.230472] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.237090] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.243363] ldlm_cn03_002 S ffff8b8bc622e180 0 117277 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.250720] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.253356] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.258550] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.265432] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.271226] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.277708] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.285279] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.290339] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.296617] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.303236] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.309515] ldlm_bl_01 S ffff8b7da96330c0 0 117278 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.316873] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.319537] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:10:51 2019][2937573.326940] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.332126] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.339182] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.344968] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.352543] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.357606] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.363879] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.371206] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.377628] ldlm_bl_02 S ffff8b7da9631040 0 117279 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.384997] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.387647] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:10:51 2019][2937573.393526] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:10:51 2019][2937573.400949] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.406159] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.413227] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.419028] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.426603] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.431668] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.438011] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.444643] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.450931] ldlm_elt S ffff8b7da9706180 0 117280 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.458321] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.460960] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:51 2019][2937573.467061] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.472257] [] expired_lock_main+0x1c2/0xc00 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.479142] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.484936] [] ? ldlm_server_glimpse_ast+0x3f0/0x3f0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.492518] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.497574] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.503850] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.510471] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.516744] mgs_params_noti S ffff8b7da9702080 0 117281 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.524108] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.526739] [] ? mgc_name2resid+0x151/0x220 [mgc] [Fri Apr 12 11:10:51 2019][2937573.533272] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:51 2019][2937573.539375] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.544535] [] mgs_ir_notify+0x1c0/0x2b0 [mgs] [Fri Apr 12 11:10:51 2019][2937573.550809] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.556586] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [Fri Apr 12 11:10:51 2019][2937573.563672] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.568733] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.575007] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.581629] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.587900] ll_mgs_0000 S ffff8b7da97030c0 0 117282 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.595280] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.597914] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.603106] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.609985] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.615841] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.622331] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.629911] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.634972] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.641245] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.647865] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.654142] ll_mgs_0001 S ffff8b7da9705140 0 117283 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.661540] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.664172] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.669370] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.676302] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.682182] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.688680] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.696257] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.701326] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.707603] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.714222] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.720491] ll_mgs_0002 S ffff8b7da9700000 0 117284 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.727868] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.730513] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.735741] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.742635] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.748431] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.754925] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.762507] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.767564] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.773840] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.780460] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.786733] ll_evictor S ffff8b8708b51040 0 117286 2 0x00000080 [Fri Apr 12 11:10:51 2019][2937573.794091] Call Trace: [Fri Apr 12 11:10:51 2019][2937573.796730] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:10:51 2019][2937573.803552] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:51 2019][2937573.809681] [] schedule+0x29/0x70 [Fri Apr 12 11:10:51 2019][2937573.814898] [] ping_evictor_main+0x4be/0x700 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.821818] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:51 2019][2937573.827664] [] ? ptlrpc_del_timeout_client+0x190/0x190 [ptlrpc] [Fri Apr 12 11:10:51 2019][2937573.835412] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:51 2019][2937573.840473] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:51 2019][2937573.846757] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:51 2019][2937573.854026] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937573.860545] ll_cfg_requeue S ffff8b8708b54100 0 117287 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937573.867919] Call Trace: [Fri Apr 12 11:10:52 2019][2937573.870559] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:52 2019][2937573.876695] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937573.881849] [] mgc_requeue_thread+0x5f5/0x870 [mgc] [Fri Apr 12 11:10:52 2019][2937573.888554] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:52 2019][2937573.894319] [] ? mgc_process_config+0x13f0/0x13f0 [mgc] [Fri Apr 12 11:10:52 2019][2937573.901373] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937573.906442] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937573.912749] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937573.919385] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937573.925684] ll_mgs_0003 S ffff8b8708b56180 0 117297 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937573.933064] Call Trace: [Fri Apr 12 11:10:52 2019][2937573.935707] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937573.940920] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937573.947805] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:52 2019][2937573.953626] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937573.960132] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937573.967721] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937573.973516] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937573.979935] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937573.986563] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937573.992833] mgs_fir_notify S ffff8b8708b52080 0 117298 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.000196] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.002869] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:52 2019][2937574.008995] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.014172] [] mgs_ir_notify+0x1c0/0x2b0 [mgs] [Fri Apr 12 11:10:52 2019][2937574.020461] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:52 2019][2937574.026252] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [Fri Apr 12 11:10:52 2019][2937574.033320] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.038414] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.044706] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.051331] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.057600] ll_mgs_0004 S ffff8b8708b50000 0 117300 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.064962] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.067596] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.072791] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.079687] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:52 2019][2937574.085501] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.091989] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.099578] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.104640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.110913] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.117533] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.123807] kmmpd-dm-3 S ffff8b8708b55140 0 117444 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.131171] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.133805] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.138951] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:52 2019][2937574.144963] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:52 2019][2937574.151510] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Fri Apr 12 11:10:52 2019][2937574.158497] [] schedule_timeout_interruptible+0x1e/0x20 [Fri Apr 12 11:10:52 2019][2937574.165558] [] kmmpd+0x216/0x430 [ldiskfs] [Fri Apr 12 11:10:52 2019][2937574.171493] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Fri Apr 12 11:10:52 2019][2937574.178199] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.183261] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.189538] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.196158] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.202431] kmmpd-dm-0 S ffff8b8cf5ed2080 0 117445 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.209804] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.212436] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.217582] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:52 2019][2937574.223596] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:52 2019][2937574.230142] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Fri Apr 12 11:10:52 2019][2937574.237108] [] schedule_timeout_interruptible+0x1e/0x20 [Fri Apr 12 11:10:52 2019][2937574.244173] [] kmmpd+0x216/0x430 [ldiskfs] [Fri Apr 12 11:10:52 2019][2937574.250109] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Fri Apr 12 11:10:52 2019][2937574.256818] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.261877] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.268154] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.274773] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.281049] jbd2/dm-0-8 S ffff8b8cf5ed30c0 0 117446 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.288412] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.291044] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.296208] [] kjournald2+0x203/0x260 [jbd2] [Fri Apr 12 11:10:52 2019][2937574.302310] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:52 2019][2937574.308329] [] ? commit_timeout+0x10/0x10 [jbd2] [Fri Apr 12 11:10:52 2019][2937574.314771] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.319833] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.326109] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.332730] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.339008] jbd2/dm-3-8 S ffff8b8cf5ed6180 0 117447 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.346366] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.349000] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.354166] [] kjournald2+0x203/0x260 [jbd2] [Fri Apr 12 11:10:52 2019][2937574.360267] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:10:52 2019][2937574.366285] [] ? commit_timeout+0x10/0x10 [jbd2] [Fri Apr 12 11:10:52 2019][2937574.372728] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.377787] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.384063] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.390684] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.396980] ldiskfs-rsv-con S ffff8b8cf5ed4100 0 117448 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.404345] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.406979] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.412128] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:52 2019][2937574.417969] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:52 2019][2937574.424156] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.429218] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.435494] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.442115] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.448387] ldiskfs-rsv-con S ffff8b8cf5ed5140 0 117449 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.455748] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.458382] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.463537] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:10:52 2019][2937574.469372] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:10:52 2019][2937574.475560] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.480621] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.486898] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.493517] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.499791] mdt00_000 S ffff8b7da966b0c0 0 117463 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.507145] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.509782] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:52 2019][2937574.516695] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.524445] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.529641] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.536819] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.543903] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:52 2019][2937574.551044] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:52 2019][2937574.556836] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.564205] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.573665] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:52 2019][2937574.580763] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:10:52 2019][2937574.587755] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:52 2019][2937574.594504] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.602558] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.609690] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.616779] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:10:52 2019][2937574.623941] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:10:52 2019][2937574.630577] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:10:52 2019][2937574.637870] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.644578] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.651695] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.658852] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:10:52 2019][2937574.665568] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:10:52 2019][2937574.672221] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.679113] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:10:52 2019][2937574.685939] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.692828] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:10:52 2019][2937574.700153] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:10:52 2019][2937574.706724] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.714013] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.721724] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.728044] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.735169] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.742953] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:52 2019][2937574.750244] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.758136] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.765119] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:52 2019][2937574.772073] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:52 2019][2937574.778214] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.784714] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.792314] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:52 2019][2937574.797370] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.803668] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:52 2019][2937574.810534] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:52 2019][2937574.816948] mdt00_001 S ffff8b5ada861040 0 117464 2 0x00000080 [Fri Apr 12 11:10:52 2019][2937574.824347] Call Trace: [Fri Apr 12 11:10:52 2019][2937574.827042] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.834815] [] schedule+0x29/0x70 [Fri Apr 12 11:10:52 2019][2937574.840009] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.850544] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:52 2019][2937574.857913] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:52 2019][2937574.863776] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937574.871033] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:10:53 2019][2937574.878380] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937574.885211] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937574.893249] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:10:53 2019][2937574.899893] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:10:53 2019][2937574.906884] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:10:53 2019][2937574.913438] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:10:53 2019][2937574.920533] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:10:53 2019][2937574.928226] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:10:53 2019][2937574.935548] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:10:53 2019][2937574.941851] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:10:53 2019][2937574.949267] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:10:53 2019][2937574.956170] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:10:53 2019][2937574.962595] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937574.969765] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:10:53 2019][2937574.976742] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:10:53 2019][2937574.983396] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:10:53 2019][2937574.991135] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:10:53 2019][2937574.998546] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:10:53 2019][2937575.004589] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:10:53 2019][2937575.011213] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:53 2019][2937575.017453] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.024187] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.031169] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:53 2019][2937575.037087] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.044183] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.051972] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:53 2019][2937575.059264] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.067138] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.074116] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:53 2019][2937575.080561] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:53 2019][2937575.086644] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.093130] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.100705] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:53 2019][2937575.105782] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:53 2019][2937575.112075] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:53 2019][2937575.118693] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:53 2019][2937575.124965] mdt00_002 S ffff8b5ada865140 0 117465 2 0x00000080 [Fri Apr 12 11:10:53 2019][2937575.132541] Call Trace: [Fri Apr 12 11:10:53 2019][2937575.135762] [] schedule+0x29/0x70 [Fri Apr 12 11:10:53 2019][2937575.141195] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.148076] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:53 2019][2937575.153881] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.160372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.168460] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:53 2019][2937575.173689] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:53 2019][2937575.179963] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:53 2019][2937575.186619] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:53 2019][2937575.192906] mdt01_000 S ffff8b5ada866180 0 117466 2 0x00000080 [Fri Apr 12 11:10:53 2019][2937575.200307] Call Trace: [Fri Apr 12 11:10:53 2019][2937575.202993] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.210801] [] schedule+0x29/0x70 [Fri Apr 12 11:10:53 2019][2937575.215981] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.223067] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.230154] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:53 2019][2937575.235951] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.243236] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:10:53 2019][2937575.251372] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.258229] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.266251] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:10:53 2019][2937575.272897] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:10:53 2019][2937575.279663] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:10:53 2019][2937575.286076] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:10:53 2019][2937575.293189] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:10:53 2019][2937575.300905] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:10:53 2019][2937575.308235] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:10:53 2019][2937575.314520] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:10:53 2019][2937575.321937] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:10:53 2019][2937575.328844] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.335252] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.342412] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.349394] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.356045] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:10:53 2019][2937575.363735] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:10:53 2019][2937575.370020] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:10:53 2019][2937575.375954] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:10:53 2019][2937575.382503] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:53 2019][2937575.388704] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.395424] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.402414] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:53 2019][2937575.408320] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.415454] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.423214] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:53 2019][2937575.430483] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.438369] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.445353] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:53 2019][2937575.451826] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:53 2019][2937575.457882] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.464368] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.471940] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:53 2019][2937575.477001] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:53 2019][2937575.483287] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:53 2019][2937575.489908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:53 2019][2937575.496199] mdt01_001 S ffff8b5ada862080 0 117467 2 0x00000080 [Fri Apr 12 11:10:53 2019][2937575.503568] Call Trace: [Fri Apr 12 11:10:53 2019][2937575.506209] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:53 2019][2937575.513144] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.520893] [] schedule+0x29/0x70 [Fri Apr 12 11:10:53 2019][2937575.526077] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.533161] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.540277] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:53 2019][2937575.547423] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:53 2019][2937575.553240] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.560596] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.568630] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.575354] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:10:53 2019][2937575.582351] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.589119] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.597173] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.604283] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.611359] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.618508] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.625136] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:10:53 2019][2937575.632327] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.638989] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.646085] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.653235] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:10:53 2019][2937575.659960] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:10:53 2019][2937575.666600] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:10:53 2019][2937575.673447] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.680336] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:10:53 2019][2937575.687672] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:10:53 2019][2937575.694269] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.701771] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.709502] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.715826] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.722933] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.730687] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:53 2019][2937575.737957] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.745834] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.752800] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:53 2019][2937575.759246] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:53 2019][2937575.765309] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.771808] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.779389] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:53 2019][2937575.784450] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:53 2019][2937575.790726] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:53 2019][2937575.797345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:53 2019][2937575.803618] mdt01_002 S ffff8b5ada864100 0 117468 2 0x00000080 [Fri Apr 12 11:10:53 2019][2937575.810999] Call Trace: [Fri Apr 12 11:10:53 2019][2937575.813665] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:53 2019][2937575.820612] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.828367] [] schedule+0x29/0x70 [Fri Apr 12 11:10:53 2019][2937575.833551] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.840633] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:53 2019][2937575.847718] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:54 2019][2937575.854857] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:54 2019][2937575.860648] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.868018] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.876049] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:54 2019][2937575.882776] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:10:54 2019][2937575.889779] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:54 2019][2937575.896535] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.904652] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.911728] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:10:54 2019][2937575.918915] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.926330] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:10:54 2019][2937575.933429] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.940093] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.947188] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.954340] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:10:54 2019][2937575.961056] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:10:54 2019][2937575.967688] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:10:54 2019][2937575.974514] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937575.981421] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:10:54 2019][2937575.988745] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:10:54 2019][2937575.995317] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.002583] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.010291] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.016620] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.023710] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.031469] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:54 2019][2937576.038739] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.046625] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.053592] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:54 2019][2937576.060057] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:54 2019][2937576.066127] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.072634] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.080224] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:54 2019][2937576.085324] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.091625] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:54 2019][2937576.098250] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.104556] mdt02_000 S ffff8b5ada8630c0 0 117469 2 0x00000080 [Fri Apr 12 11:10:54 2019][2937576.111920] Call Trace: [Fri Apr 12 11:10:54 2019][2937576.114557] [] schedule+0x29/0x70 [Fri Apr 12 11:10:54 2019][2937576.119763] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.126663] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:54 2019][2937576.132455] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.138942] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.146517] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:54 2019][2937576.151576] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.157854] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:54 2019][2937576.164474] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.170750] mdt02_001 S ffff8b5cfa165140 0 117470 2 0x00000080 [Fri Apr 12 11:10:54 2019][2937576.178129] Call Trace: [Fri Apr 12 11:10:54 2019][2937576.180776] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:54 2019][2937576.187700] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.195449] [] schedule+0x29/0x70 [Fri Apr 12 11:10:54 2019][2937576.200632] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.207718] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.214819] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:54 2019][2937576.221988] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:54 2019][2937576.227780] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.235126] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.243169] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.250936] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:10:54 2019][2937576.257854] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.265643] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.273671] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:10:54 2019][2937576.280303] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:54 2019][2937576.286507] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:54 2019][2937576.293234] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:10:54 2019][2937576.300214] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:54 2019][2937576.306111] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.313205] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.320981] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:54 2019][2937576.328260] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.336137] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.343109] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:54 2019][2937576.349567] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:54 2019][2937576.355638] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.362154] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.369761] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:54 2019][2937576.374837] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.381126] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:54 2019][2937576.387750] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.394024] mdt02_002 S ffff8b58dce65140 0 117471 2 0x00000080 [Fri Apr 12 11:10:54 2019][2937576.401386] Call Trace: [Fri Apr 12 11:10:54 2019][2937576.404016] [] schedule+0x29/0x70 [Fri Apr 12 11:10:54 2019][2937576.409165] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:54 2019][2937576.415183] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:54 2019][2937576.421767] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.429560] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.436663] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.443759] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:54 2019][2937576.450916] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:54 2019][2937576.456723] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.464096] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.472138] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.479903] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:10:54 2019][2937576.486395] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.494176] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.502199] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:10:54 2019][2937576.508847] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:10:54 2019][2937576.516530] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:10:54 2019][2937576.522814] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:10:54 2019][2937576.528755] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:10:54 2019][2937576.535320] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:54 2019][2937576.541528] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:54 2019][2937576.548262] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:10:54 2019][2937576.555238] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:54 2019][2937576.561132] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.568224] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.576003] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:54 2019][2937576.583271] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.591148] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.598130] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:54 2019][2937576.604583] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:54 2019][2937576.610642] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.617125] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.624700] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:54 2019][2937576.629792] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.636084] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:54 2019][2937576.642701] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.648975] mdt03_000 S ffff8b58dce62080 0 117472 2 0x00000080 [Fri Apr 12 11:10:54 2019][2937576.656348] Call Trace: [Fri Apr 12 11:10:54 2019][2937576.658981] [] schedule+0x29/0x70 [Fri Apr 12 11:10:54 2019][2937576.664171] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.671072] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:54 2019][2937576.676862] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.683342] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.690915] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:54 2019][2937576.695987] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.702293] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:54 2019][2937576.708919] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.715190] mdt03_001 S ffff8b58dce61040 0 117473 2 0x00000080 [Fri Apr 12 11:10:54 2019][2937576.722562] Call Trace: [Fri Apr 12 11:10:54 2019][2937576.725196] [] schedule+0x29/0x70 [Fri Apr 12 11:10:54 2019][2937576.730425] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.737320] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:10:54 2019][2937576.744290] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:54 2019][2937576.750085] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.756582] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.764169] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:54 2019][2937576.769231] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.775505] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:54 2019][2937576.782126] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:54 2019][2937576.788400] mdt03_002 S ffff8b58dce66180 0 117474 2 0x00000080 [Fri Apr 12 11:10:54 2019][2937576.795764] Call Trace: [Fri Apr 12 11:10:54 2019][2937576.798420] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:54 2019][2937576.805368] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.813122] [] schedule+0x29/0x70 [Fri Apr 12 11:10:54 2019][2937576.818312] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.825397] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.832913] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:54 2019][2937576.840093] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:54 2019][2937576.845886] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:54 2019][2937576.853239] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.861281] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.869056] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:10:55 2019][2937576.875974] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.883752] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.891776] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:10:55 2019][2937576.898414] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:55 2019][2937576.904608] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:55 2019][2937576.911326] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:10:55 2019][2937576.918302] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:55 2019][2937576.924198] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.931318] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.939083] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:55 2019][2937576.946387] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.954263] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.961228] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:55 2019][2937576.967675] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:55 2019][2937576.973728] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.980208] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937576.987783] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937576.992844] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937576.999138] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.005756] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.012028] mdt_rdpg00_000 S ffff8b58dce630c0 0 117475 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.019384] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.022019] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.027226] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.034108] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.039899] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.046396] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.053981] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.059042] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.065317] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.071937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.078210] mdt_rdpg00_001 S ffff8b58dce64100 0 117476 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.085590] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.088235] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.093424] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.100307] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.106094] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.112578] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.120151] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.125213] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.131499] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.138119] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.144387] mdt_rdpg01_000 S ffff8b536762a080 0 117477 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.151746] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.154377] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.159564] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.166440] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.172235] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.178716] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.186286] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.191352] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.197634] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.204265] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.210535] mdt_rdpg01_001 D ffff8b536762c100 0 117478 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.217900] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.220531] [] schedule_preempt_disabled+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.227152] [] ? wake_up_q+0x5b/0x80 [Fri Apr 12 11:10:55 2019][2937577.232560] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:10:55 2019][2937577.238853] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:10:55 2019][2937577.244801] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:10:55 2019][2937577.250555] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:10:55 2019][2937577.257174] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:10:55 2019][2937577.263361] [] ? schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.268730] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.275780] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.281576] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.288229] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.295805] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.301039] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.307325] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.314134] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.320406] mdt_rdpg02_000 S ffff8b536762e180 0 117479 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.327788] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.330420] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.335646] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.342528] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.348320] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.354811] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.362392] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.367452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.373728] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.380347] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.386621] mdt_rdpg02_001 S ffff8b536762d140 0 117480 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.393983] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.396619] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.401816] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.408698] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.414491] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.420970] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.428553] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.433616] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.439900] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.446537] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.452813] mdt_rdpg03_000 S ffff8b5367629040 0 117481 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.460174] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.462815] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.468003] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.474887] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.480707] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.487196] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.494769] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.499829] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.506115] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.512736] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.519005] mdt_rdpg03_001 S ffff8b536762b0c0 0 117482 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.526364] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.528993] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.534183] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.541068] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.546861] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.553351] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.560933] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.566026] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.572304] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.578933] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.585204] mdt_attr00_000 S ffff8b5cf32dc100 0 117483 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.592569] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.595203] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.600409] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.607291] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.613091] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.619582] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.627156] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.632216] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.638501] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.645122] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.651420] mdt_attr00_001 S ffff8b5cf32db0c0 0 117484 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.658809] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.661442] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.666635] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.673531] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.679324] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.685831] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.693405] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.698465] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.704742] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.711370] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.717642] mdt_attr01_000 S ffff8b5cf32da080 0 117485 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.725042] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.727691] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.732881] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.739781] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.745575] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.752071] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.759664] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.764726] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.771000] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.777630] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.783911] mdt_attr01_001 S ffff8b5cf32d8000 0 117486 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.791275] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.793917] [] schedule+0x29/0x70 [Fri Apr 12 11:10:55 2019][2937577.799112] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.805997] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:55 2019][2937577.811788] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.818283] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:55 2019][2937577.825860] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:55 2019][2937577.830937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.837221] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:55 2019][2937577.843852] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:55 2019][2937577.850133] mdt_attr02_000 S ffff8b5cf32d9040 0 117487 2 0x00000080 [Fri Apr 12 11:10:55 2019][2937577.857524] Call Trace: [Fri Apr 12 11:10:55 2019][2937577.860182] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937577.865404] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937577.872315] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937577.878111] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937577.884598] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937577.892176] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937577.897240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937577.903519] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937577.910154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937577.916429] mdt_attr02_001 S ffff8b5cb6b20000 0 117488 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937577.923809] Call Trace: [Fri Apr 12 11:10:56 2019][2937577.926448] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937577.931642] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937577.938528] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937577.944335] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937577.950830] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937577.958402] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937577.963465] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937577.969740] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937577.976361] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937577.982643] mdt_attr03_000 S ffff8b5cb6b21040 0 117489 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937577.990014] Call Trace: [Fri Apr 12 11:10:56 2019][2937577.992656] [] ? lc_watchdog_add+0x5e/0x790 [libcfs] [Fri Apr 12 11:10:56 2019][2937577.999449] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.004641] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.011530] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.017323] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.023824] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.031395] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.036457] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.042740] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.049370] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.055659] mdt_attr03_001 S ffff8b5cb6b24100 0 117490 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.063024] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.065655] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.070844] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.077729] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.083519] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.090020] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.097593] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.102653] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.108937] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.115762] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.122056] mdt00_003 S ffff8b5cb6b26180 0 117491 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.129418] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.132088] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.139834] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.145032] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.152121] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.159179] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.164969] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.172223] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:10:56 2019][2937578.179569] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.186400] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.194440] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:10:56 2019][2937578.201072] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:10:56 2019][2937578.207791] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:10:56 2019][2937578.214181] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:10:56 2019][2937578.221239] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:10:56 2019][2937578.228928] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:10:56 2019][2937578.236253] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:10:56 2019][2937578.242542] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:10:56 2019][2937578.249961] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:10:56 2019][2937578.256895] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:10:56 2019][2937578.263320] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.270480] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:10:56 2019][2937578.277457] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:10:56 2019][2937578.284110] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:10:56 2019][2937578.291796] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:10:56 2019][2937578.298082] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:10:56 2019][2937578.304019] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:10:56 2019][2937578.310569] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:56 2019][2937578.316769] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:56 2019][2937578.323487] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:10:56 2019][2937578.330461] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:56 2019][2937578.336356] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.343451] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.351224] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:56 2019][2937578.358543] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.366413] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.373384] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:56 2019][2937578.379838] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:56 2019][2937578.385891] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.392386] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.399979] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.405037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.411352] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.417973] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.424243] mdt_out00_000 S ffff8b5cb6b22080 0 117492 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.431610] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.434249] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.439440] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.446329] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.452150] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.458642] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.466235] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.471295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.477572] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.484204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.490474] mdt_out00_001 S ffff8b5cb6b230c0 0 117493 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.497829] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.500464] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.505658] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.512535] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:10:56 2019][2937578.519501] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.525293] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.531777] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.539351] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.544433] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.550706] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.557338] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.563606] mdt_out01_000 S ffff8b5cf8784100 0 117494 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.570971] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.573602] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.578809] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.585695] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.591521] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.598020] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.605591] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.610652] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.616948] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.623586] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.629866] mdt_out01_001 S ffff8b5cf8781040 0 117495 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.637227] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.639859] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.645051] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.651934] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.657735] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.664221] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.671799] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.676860] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.683140] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.689766] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.696041] mdt_out02_000 S ffff8b5cf8785140 0 117496 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.703427] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.706058] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.711250] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.718133] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.723927] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.730414] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.737987] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.743047] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.749333] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.755953] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.762236] mdt_out02_001 S ffff8b5cf8782080 0 117497 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.769621] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.772266] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.777460] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.784348] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.790145] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.796639] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.804220] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:56 2019][2937578.809280] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.815554] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:56 2019][2937578.822176] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:56 2019][2937578.828450] mdt_out03_000 S ffff8b5cf8780000 0 117498 2 0x00000080 [Fri Apr 12 11:10:56 2019][2937578.835816] Call Trace: [Fri Apr 12 11:10:56 2019][2937578.838455] [] schedule+0x29/0x70 [Fri Apr 12 11:10:56 2019][2937578.843667] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.850563] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:56 2019][2937578.856363] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:56 2019][2937578.862860] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937578.870443] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937578.875512] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937578.881788] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937578.888409] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937578.894680] mdt_out03_001 S ffff8b5cf87830c0 0 117499 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937578.902054] Call Trace: [Fri Apr 12 11:10:57 2019][2937578.904703] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937578.909912] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937578.916796] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937578.922605] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937578.929093] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937578.936675] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937578.941737] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937578.948015] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937578.954645] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937578.960917] mdt_seqs_0000 S ffff8b4e2965b0c0 0 117500 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937578.968276] Call Trace: [Fri Apr 12 11:10:57 2019][2937578.970909] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937578.976102] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937578.982982] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937578.988777] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937578.995267] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.002847] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.007907] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.014181] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.020803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.027076] mdt_seqs_0001 S ffff8b538323c100 0 117501 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.034437] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.037069] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.042261] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.049145] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.054935] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.061440] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.069043] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.074114] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.080398] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.087037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.093310] mdt_seqm_0000 S ffff8b5383239040 0 117502 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.100690] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.103321] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.108511] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.115394] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.121184] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.127670] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.135241] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.140301] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.146577] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.153223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.159496] mdt_seqm_0001 S ffff8b538323a080 0 117503 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.166860] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.169492] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.174680] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.181565] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.187358] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.193840] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.201411] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.206472] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.212751] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.219385] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.225678] mdt_fld_0000 S ffff8b538323e180 0 117504 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.233040] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.235690] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.240898] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.247791] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.253587] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.260074] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.267656] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.272713] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.278991] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.285610] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.291884] mdt_fld_0001 S ffff8b538323d140 0 117505 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.299247] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.301879] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.307072] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.313951] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.319744] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.326226] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.333798] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.338860] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.345134] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.351754] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.358031] mdt03_003 S ffff8b538323b0c0 0 117506 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.365391] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.368023] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.373216] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.380114] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.385894] [] ? class_conn2export+0x4a/0x2c0 [obdclass] [Fri Apr 12 11:10:57 2019][2937579.393074] [] ptlrpc_main+0xc7c/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.399553] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.407128] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.412188] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.418464] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.425086] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.431360] mdt_io00_000 S ffff8b5c62396180 0 117507 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.438748] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.441384] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.446576] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.453454] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.459245] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.465729] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.473299] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.478361] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.484642] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.491267] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.495057] Lustre: fir-MDT0000-osp-MDT0001: Connection to fir-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [Fri Apr 12 11:10:57 2019][2937579.495156] Lustre: Skipped 3 previous similar messages [Fri Apr 12 11:10:57 2019][2937579.518045] mdt_io00_001 S ffff8b5c623930c0 0 117508 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.525409] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.528043] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.533234] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.540114] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.545908] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.552415] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.559987] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.565066] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.571401] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.578028] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.584324] mdt_io00_002 S ffff8b5c62394100 0 117509 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.591684] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.594318] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.599511] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.606390] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.612182] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.618665] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.626244] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.631322] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.637599] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.644238] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.650510] mdt_io01_000 S ffff8b5c62395140 0 117510 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.657874] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.660505] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.665703] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.672589] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.678393] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.684902] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.692518] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.697601] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.703875] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.710495] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.716769] mdt_io01_001 S ffff8b5c62390000 0 117511 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.724131] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.726763] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.731959] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.738837] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.744630] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.751122] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.758693] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.763753] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.770026] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.776651] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.782923] mdt_io01_002 S ffff8b5cf7fb2080 0 117512 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.790284] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.792918] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.798108] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.804992] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:57 2019][2937579.810785] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.817273] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:57 2019][2937579.824847] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:57 2019][2937579.829907] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.836186] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:57 2019][2937579.842805] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:57 2019][2937579.849077] mdt_io02_000 S ffff8b5cf7fb30c0 0 117513 2 0x00000080 [Fri Apr 12 11:10:57 2019][2937579.856438] Call Trace: [Fri Apr 12 11:10:57 2019][2937579.859073] [] schedule+0x29/0x70 [Fri Apr 12 11:10:57 2019][2937579.864267] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.871143] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:58 2019][2937579.876945] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.883428] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.891002] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:58 2019][2937579.896060] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937579.902337] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:58 2019][2937579.908973] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937579.915321] mdt_io02_001 S ffff8b5cf7fb6180 0 117514 2 0x00000080 [Fri Apr 12 11:10:58 2019][2937579.922690] Call Trace: [Fri Apr 12 11:10:58 2019][2937579.925347] [] schedule+0x29/0x70 [Fri Apr 12 11:10:58 2019][2937579.930607] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.937488] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:58 2019][2937579.943283] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.949761] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.957336] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:58 2019][2937579.962396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937579.968674] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:58 2019][2937579.975305] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937579.981576] mdt_io02_002 S ffff8b5cf7fb4100 0 117515 2 0x00000080 [Fri Apr 12 11:10:58 2019][2937579.987655] LNet: Service thread pid 117550 was inactive for 200.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 11:10:58 2019][2937579.987658] LNet: Skipped 3 previous similar messages [Fri Apr 12 11:10:58 2019][2937579.987661] Pid: 117550, comm: mdt00_004 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:10:58 2019][2937579.987661] Call Trace: [Fri Apr 12 11:10:58 2019][2937579.987718] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.987747] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.987762] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:10:58 2019][2937579.987773] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:10:58 2019][2937579.987784] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:58 2019][2937579.987794] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:58 2019][2937579.987804] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:58 2019][2937579.987851] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.987885] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.987919] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937579.987924] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:58 2019][2937579.987954] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:58 2019][2937579.988082] [] 0xffffffffffffffff [Fri Apr 12 11:10:58 2019][2937579.988215] LustreError: dumping log to /tmp/lustre-log.1555092658.117550 [Fri Apr 12 11:10:58 2019][2937580.115305] Call Trace: [Fri Apr 12 11:10:58 2019][2937580.117949] [] schedule+0x29/0x70 [Fri Apr 12 11:10:58 2019][2937580.123148] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.130041] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:58 2019][2937580.135958] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.142449] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.150029] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:58 2019][2937580.155087] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.161363] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:58 2019][2937580.167983] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.174256] mdt_io03_000 S ffff8b5cf7fb5140 0 117516 2 0x00000080 [Fri Apr 12 11:10:58 2019][2937580.181620] Call Trace: [Fri Apr 12 11:10:58 2019][2937580.184254] [] schedule+0x29/0x70 [Fri Apr 12 11:10:58 2019][2937580.189444] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.196323] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:58 2019][2937580.202194] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.208700] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.216285] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:58 2019][2937580.221348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.227633] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:58 2019][2937580.234282] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.240584] mdt_io03_001 S ffff8b5cfc2b1040 0 117517 2 0x00000080 [Fri Apr 12 11:10:58 2019][2937580.247984] Call Trace: [Fri Apr 12 11:10:58 2019][2937580.250647] [] schedule+0x29/0x70 [Fri Apr 12 11:10:58 2019][2937580.255856] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.262771] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:58 2019][2937580.268571] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.275065] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.282640] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:58 2019][2937580.287700] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.293982] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:58 2019][2937580.300657] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.306941] mdt_io03_002 S ffff8b5cfc2b30c0 0 117518 2 0x00000080 [Fri Apr 12 11:10:58 2019][2937580.314332] Call Trace: [Fri Apr 12 11:10:58 2019][2937580.316971] [] schedule+0x29/0x70 [Fri Apr 12 11:10:58 2019][2937580.322166] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.329088] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:58 2019][2937580.334925] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.341440] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.349018] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:58 2019][2937580.354077] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.360354] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:58 2019][2937580.366974] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.373268] mdt01_003 S ffff8b5cfc2b5140 0 117530 2 0x00000080 [Fri Apr 12 11:10:58 2019][2937580.380630] Call Trace: [Fri Apr 12 11:10:58 2019][2937580.383267] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:58 2019][2937580.390189] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.397934] [] schedule+0x29/0x70 [Fri Apr 12 11:10:58 2019][2937580.403118] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.410201] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.417284] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:58 2019][2937580.424429] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:58 2019][2937580.430220] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.437581] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.445602] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.452333] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:10:58 2019][2937580.459349] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.466112] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.474171] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.481267] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.488337] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.495503] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.503666] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:10:58 2019][2937580.511450] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.518143] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.525249] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.532416] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:10:58 2019][2937580.539138] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:10:58 2019][2937580.545765] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.552596] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.559485] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:10:58 2019][2937580.566805] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:10:58 2019][2937580.573379] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.580677] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.588399] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.594733] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.601845] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.609604] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:58 2019][2937580.616884] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.624755] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.631725] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:58 2019][2937580.638173] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:58 2019][2937580.644231] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.650719] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.658322] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:58 2019][2937580.663395] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.669671] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:58 2019][2937580.676306] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:58 2019][2937580.682578] mdt02_003 S ffff8b5cfc2b2080 0 117531 2 0x00000080 [Fri Apr 12 11:10:58 2019][2937580.689942] Call Trace: [Fri Apr 12 11:10:58 2019][2937580.692583] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:58 2019][2937580.699502] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.707254] [] schedule+0x29/0x70 [Fri Apr 12 11:10:58 2019][2937580.712438] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.719544] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.726623] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:58 2019][2937580.733782] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:58 2019][2937580.739574] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.746922] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.754959] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.761678] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:10:58 2019][2937580.768665] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.775432] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.783458] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.790677] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.797836] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:10:58 2019][2937580.805016] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.811680] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.818789] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.825962] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:10:58 2019][2937580.832683] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:10:58 2019][2937580.839324] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:10:58 2019][2937580.846187] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:10:58 2019][2937580.853106] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:10:58 2019][2937580.860432] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:10:59 2019][2937580.867020] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.874330] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.882044] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.888360] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.895452] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.903207] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:59 2019][2937580.910469] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.918336] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.925325] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:59 2019][2937580.931766] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:59 2019][2937580.937822] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.944303] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937580.951874] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:59 2019][2937580.956934] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937580.963203] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:59 2019][2937580.969826] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937580.976099] mdt02_004 S ffff8b5cfc25e180 0 117532 2 0x00000080 [Fri Apr 12 11:10:59 2019][2937580.983471] Call Trace: [Fri Apr 12 11:10:59 2019][2937580.986107] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:10:59 2019][2937580.992989] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:10:59 2019][2937580.999128] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.006875] [] schedule+0x29/0x70 [Fri Apr 12 11:10:59 2019][2937581.012069] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.019172] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.026255] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:59 2019][2937581.033397] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:59 2019][2937581.039184] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.046533] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.054581] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.062339] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:10:59 2019][2937581.068815] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.076598] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.084622] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:10:59 2019][2937581.091269] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:10:59 2019][2937581.098950] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:10:59 2019][2937581.105237] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:10:59 2019][2937581.111174] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:10:59 2019][2937581.117719] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:59 2019][2937581.123913] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:59 2019][2937581.130631] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:10:59 2019][2937581.137608] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:59 2019][2937581.143498] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.150596] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.158354] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:59 2019][2937581.165638] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.173516] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.180482] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:59 2019][2937581.186929] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:59 2019][2937581.192999] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.199481] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.207052] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:59 2019][2937581.212113] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.218390] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:59 2019][2937581.225011] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.231286] mdt02_005 S ffff8b5cfc25a080 0 117533 2 0x00000080 [Fri Apr 12 11:10:59 2019][2937581.238649] Call Trace: [Fri Apr 12 11:10:59 2019][2937581.241278] [] schedule+0x29/0x70 [Fri Apr 12 11:10:59 2019][2937581.246468] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.253343] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:59 2019][2937581.259122] [] ? class_conn2export+0x4a/0x2c0 [obdclass] [Fri Apr 12 11:10:59 2019][2937581.266302] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.272785] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.280357] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:59 2019][2937581.285417] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.291696] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:59 2019][2937581.298314] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.304597] mdt01_004 S ffff8b5cfc259040 0 117534 2 0x00000080 [Fri Apr 12 11:10:59 2019][2937581.311954] Call Trace: [Fri Apr 12 11:10:59 2019][2937581.314583] [] schedule+0x29/0x70 [Fri Apr 12 11:10:59 2019][2937581.319770] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.326647] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:59 2019][2937581.332441] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.338924] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.346502] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:59 2019][2937581.351564] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.357840] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:59 2019][2937581.364458] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.370724] mdt02_006 S ffff8b4cecbcb0c0 0 117535 2 0x00000080 [Fri Apr 12 11:10:59 2019][2937581.378097] Call Trace: [Fri Apr 12 11:10:59 2019][2937581.380729] [] ? ___slab_alloc+0x3ac/0x4f0 [Fri Apr 12 11:10:59 2019][2937581.386660] [] schedule+0x29/0x70 [Fri Apr 12 11:10:59 2019][2937581.391803] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:10:59 2019][2937581.397819] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:10:59 2019][2937581.404395] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.412171] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.419262] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.426343] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:10:59 2019][2937581.433505] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:59 2019][2937581.439324] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.446678] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.454729] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.462491] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:10:59 2019][2937581.468976] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.476793] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.484816] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:10:59 2019][2937581.491518] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:10:59 2019][2937581.499214] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:10:59 2019][2937581.505528] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:10:59 2019][2937581.511463] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:10:59 2019][2937581.518008] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:10:59 2019][2937581.524214] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:10:59 2019][2937581.531896] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:10:59 2019][2937581.538891] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:10:59 2019][2937581.544793] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.551957] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.559727] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:10:59 2019][2937581.567100] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.574984] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.581953] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:10:59 2019][2937581.588445] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:10:59 2019][2937581.594517] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.601051] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.608643] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:59 2019][2937581.613705] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.620002] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:59 2019][2937581.626645] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.632952] mdt02_007 S ffff8b4cecbce180 0 117536 2 0x00000080 [Fri Apr 12 11:10:59 2019][2937581.640332] Call Trace: [Fri Apr 12 11:10:59 2019][2937581.642964] [] schedule+0x29/0x70 [Fri Apr 12 11:10:59 2019][2937581.648153] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.655047] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:59 2019][2937581.660856] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.667379] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.674979] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:59 2019][2937581.680043] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.686316] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:59 2019][2937581.692948] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.699226] osp_up0-2 S ffff8b4cecbcc100 0 117546 2 0x00000080 [Fri Apr 12 11:10:59 2019][2937581.706687] Call Trace: [Fri Apr 12 11:10:59 2019][2937581.709340] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:10:59 2019][2937581.716323] [] schedule+0x29/0x70 [Fri Apr 12 11:10:59 2019][2937581.721481] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:10:59 2019][2937581.728590] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:59 2019][2937581.734433] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:10:59 2019][2937581.741689] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:59 2019][2937581.746748] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.753042] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:59 2019][2937581.759682] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.765962] osp-pre-0-2 S ffff8b4cecbc9040 0 117547 2 0x00000080 [Fri Apr 12 11:10:59 2019][2937581.773334] Call Trace: [Fri Apr 12 11:10:59 2019][2937581.775988] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:10:59 2019][2937581.781420] [] schedule+0x29/0x70 [Fri Apr 12 11:10:59 2019][2937581.786580] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:10:59 2019][2937581.793583] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:10:59 2019][2937581.799371] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:10:59 2019][2937581.806090] [] kthread+0xd1/0xe0 [Fri Apr 12 11:10:59 2019][2937581.811153] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.817425] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:10:59 2019][2937581.824049] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:10:59 2019][2937581.830331] mdt03_004 S ffff8b4cecbc8000 0 117548 2 0x00000080 [Fri Apr 12 11:10:59 2019][2937581.837694] Call Trace: [Fri Apr 12 11:10:59 2019][2937581.840370] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.848117] [] schedule+0x29/0x70 [Fri Apr 12 11:10:59 2019][2937581.853341] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.860433] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:10:59 2019][2937581.867534] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:00 2019][2937581.873443] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937581.880841] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:00 2019][2937581.888281] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937581.895135] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937581.903165] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:00 2019][2937581.909807] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:00 2019][2937581.916524] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:00 2019][2937581.922978] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:00 2019][2937581.930067] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:00 2019][2937581.937764] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:00 2019][2937581.945093] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:00 2019][2937581.951415] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:00 2019][2937581.958848] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:00 2019][2937581.965755] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:00 2019][2937581.972162] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937581.979348] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:00 2019][2937581.986356] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:00 2019][2937581.993017] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.000708] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.007000] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:00 2019][2937582.012934] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:00 2019][2937582.019526] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:00 2019][2937582.025771] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.032532] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.039570] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:00 2019][2937582.045571] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.052740] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.060604] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:00 2019][2937582.067938] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.075824] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.082792] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:00 2019][2937582.089237] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:00 2019][2937582.095308] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.101788] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.109361] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:00 2019][2937582.114420] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:00 2019][2937582.120693] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:00 2019][2937582.127313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:00 2019][2937582.133594] mdt03_005 S ffff8b4cecbca080 0 117549 2 0x00000080 [Fri Apr 12 11:11:00 2019][2937582.141003] Call Trace: [Fri Apr 12 11:11:00 2019][2937582.143658] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:00 2019][2937582.150575] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.158340] [] schedule+0x29/0x70 [Fri Apr 12 11:11:00 2019][2937582.163520] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.170608] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.177705] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.184878] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:00 2019][2937582.190706] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.198068] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.206086] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.212850] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:00 2019][2937582.219849] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.226603] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.234650] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.241846] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.249028] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:11:00 2019][2937582.256268] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.262936] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.270041] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.277191] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:00 2019][2937582.283913] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:00 2019][2937582.290575] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.297472] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.304306] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.311199] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:00 2019][2937582.318517] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:00 2019][2937582.325088] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.332368] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.340089] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.346410] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.353508] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.361266] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:00 2019][2937582.368574] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.376456] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.383456] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:00 2019][2937582.389930] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:00 2019][2937582.396010] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.402497] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.410076] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:00 2019][2937582.415140] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:00 2019][2937582.421451] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:00 2019][2937582.428068] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:00 2019][2937582.434369] mdt00_004 S ffff8b5cfa34c100 0 117550 2 0x00000080 [Fri Apr 12 11:11:00 2019][2937582.441731] Call Trace: [Fri Apr 12 11:11:00 2019][2937582.444380] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:00 2019][2937582.451260] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:00 2019][2937582.457400] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.465146] [] schedule+0x29/0x70 [Fri Apr 12 11:11:00 2019][2937582.470326] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.477412] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.484490] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.491629] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:00 2019][2937582.497442] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.504785] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.512830] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.520593] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.527139] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.534955] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.543006] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.549679] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.557406] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.563707] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:00 2019][2937582.569659] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:00 2019][2937582.576207] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:00 2019][2937582.582399] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.589116] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.596094] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:00 2019][2937582.601988] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.609084] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.616840] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:00 2019][2937582.624134] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.632019] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.638988] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:00 2019][2937582.645473] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:00 2019][2937582.651588] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.658072] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.665646] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:00 2019][2937582.670720] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:00 2019][2937582.677010] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:00 2019][2937582.683643] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:00 2019][2937582.689920] mdt00_005 S ffff8b5cfa34e180 0 117551 2 0x00000080 [Fri Apr 12 11:11:00 2019][2937582.697283] Call Trace: [Fri Apr 12 11:11:00 2019][2937582.699921] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:00 2019][2937582.706823] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:00 2019][2937582.712961] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.720716] [] schedule+0x29/0x70 [Fri Apr 12 11:11:00 2019][2937582.725912] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.733055] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.740158] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.747306] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:00 2019][2937582.753093] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.760443] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.768521] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.776311] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.782793] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.790595] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:00 2019][2937582.798627] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.805296] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.812999] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:00 2019][2937582.819287] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:00 2019][2937582.825229] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:00 2019][2937582.831785] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:00 2019][2937582.838006] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.844725] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:00 2019][2937582.851702] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:00 2019][2937582.857599] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.864726] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.872505] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:01 2019][2937582.879785] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.887662] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.894633] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:01 2019][2937582.901082] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:01 2019][2937582.907169] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.913669] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.921276] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:01 2019][2937582.926344] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937582.932620] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:01 2019][2937582.939268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937582.945579] mdt01_005 S ffff8b5cfa349040 0 117552 2 0x00000080 [Fri Apr 12 11:11:01 2019][2937582.952973] Call Trace: [Fri Apr 12 11:11:01 2019][2937582.955625] [] schedule+0x29/0x70 [Fri Apr 12 11:11:01 2019][2937582.963619] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.970547] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:01 2019][2937582.976354] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.982855] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937582.990455] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:01 2019][2937582.995542] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937583.001825] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:01 2019][2937583.008465] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937583.014744] mdt01_006 S ffff8b5cfa34b0c0 0 117557 2 0x00000080 [Fri Apr 12 11:11:01 2019][2937583.022127] Call Trace: [Fri Apr 12 11:11:01 2019][2937583.024773] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:01 2019][2937583.031724] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.039483] [] schedule+0x29/0x70 [Fri Apr 12 11:11:01 2019][2937583.044675] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.051766] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.058874] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:01 2019][2937583.066041] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:01 2019][2937583.071842] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.079191] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.087215] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.093938] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:01 2019][2937583.100939] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.107695] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.115752] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.122845] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.129918] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.137092] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.143722] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:01 2019][2937583.150928] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.157601] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.164755] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.171923] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:01 2019][2937583.178641] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:01 2019][2937583.185297] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.192124] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.199022] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:01 2019][2937583.206343] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:01 2019][2937583.212926] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.220200] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.227939] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.234279] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.241393] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.249169] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:01 2019][2937583.256437] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.264318] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.271306] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:01 2019][2937583.277751] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:01 2019][2937583.283816] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.290333] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.297922] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:01 2019][2937583.303000] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937583.309271] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:01 2019][2937583.315894] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937583.322178] mdt02_008 S ffff8b5cfa34a080 0 117558 2 0x00000080 [Fri Apr 12 11:11:01 2019][2937583.329551] Call Trace: [Fri Apr 12 11:11:01 2019][2937583.332189] [] schedule+0x29/0x70 [Fri Apr 12 11:11:01 2019][2937583.337346] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:11:01 2019][2937583.343390] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:01 2019][2937583.350709] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:11:01 2019][2937583.357249] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Fri Apr 12 11:11:01 2019][2937583.364199] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.372019] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.379106] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.386163] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:01 2019][2937583.391955] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.399218] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:01 2019][2937583.406565] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.413393] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.421426] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:01 2019][2937583.428048] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:01 2019][2937583.434761] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:01 2019][2937583.441150] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:01 2019][2937583.448209] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:01 2019][2937583.455891] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:01 2019][2937583.463211] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:01 2019][2937583.469499] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:01 2019][2937583.476907] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:01 2019][2937583.483800] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.490204] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.497355] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.504332] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.510977] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:01 2019][2937583.518663] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:01 2019][2937583.524947] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:01 2019][2937583.530883] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:01 2019][2937583.537432] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:01 2019][2937583.543623] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.550348] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.557330] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:01 2019][2937583.563221] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.570317] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.578089] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:01 2019][2937583.585362] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.593236] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.600202] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:01 2019][2937583.606650] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:01 2019][2937583.612722] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.619217] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.626793] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:01 2019][2937583.631851] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937583.638128] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:01 2019][2937583.644748] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937583.651022] mdt03_006 S ffff8b5cfa348000 0 117559 2 0x00000080 [Fri Apr 12 11:11:01 2019][2937583.658385] Call Trace: [Fri Apr 12 11:11:01 2019][2937583.661017] [] schedule+0x29/0x70 [Fri Apr 12 11:11:01 2019][2937583.666206] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.673091] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:01 2019][2937583.678884] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.685367] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.692945] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:01 2019][2937583.698006] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937583.704282] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:01 2019][2937583.710902] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:01 2019][2937583.717174] mdt00_006 S ffff8b4e8c19c100 0 117560 2 0x00000080 [Fri Apr 12 11:11:01 2019][2937583.724531] Call Trace: [Fri Apr 12 11:11:01 2019][2937583.727167] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:01 2019][2937583.734088] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.741835] [] schedule+0x29/0x70 [Fri Apr 12 11:11:01 2019][2937583.747015] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.754104] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.761185] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:01 2019][2937583.768330] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:01 2019][2937583.774118] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.781469] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.789512] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.797278] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:01 2019][2937583.804194] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.811977] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.819997] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:01 2019][2937583.826626] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:01 2019][2937583.832823] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.839540] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:01 2019][2937583.846520] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:01 2019][2937583.852417] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:01 2019][2937583.859510] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:02 2019][2937583.867258] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:02 2019][2937583.874520] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:02 2019][2937583.882386] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:02 2019][2937583.889353] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:02 2019][2937583.895800] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:02 2019][2937583.901858] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:02 2019][2937583.908346] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:02 2019][2937583.915917] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937583.920978] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937583.927253] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937583.933873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937583.940148] osp_up1-2 S ffff8b4e8c19a080 0 117564 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937583.947511] Call Trace: [Fri Apr 12 11:11:02 2019][2937583.950150] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:11:02 2019][2937583.957119] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937583.962273] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:11:02 2019][2937583.969321] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937583.975081] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:11:02 2019][2937583.982306] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937583.987365] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937583.993641] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.000263] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.006537] osp-pre-1-2 S ffff8b4e8c19e180 0 117565 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.013901] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.016531] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:02 2019][2937584.021939] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.027092] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:02 2019][2937584.034056] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937584.039812] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:02 2019][2937584.046517] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937584.051579] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.057854] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.064474] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.070750] osp_up3-2 S ffff8b4e8c199040 0 117566 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.078130] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.080768] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:11:02 2019][2937584.087740] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.092891] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:11:02 2019][2937584.099942] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937584.105700] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:11:02 2019][2937584.112921] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937584.117981] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.124251] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.130870] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.137136] osp-pre-3-2 S ffff8b4e8c19b0c0 0 117567 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.144491] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.147123] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:02 2019][2937584.152529] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.157684] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:02 2019][2937584.164647] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937584.170407] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:02 2019][2937584.177109] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937584.182174] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.188467] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.195084] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.201359] osp-syn-12-2 S ffff8b4e8c198000 0 117568 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.208723] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.211362] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:02 2019][2937584.217464] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.222617] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:02 2019][2937584.229926] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937584.235701] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.243019] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:02 2019][2937584.249605] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.256867] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.264119] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.271450] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.279215] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.286461] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.294237] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.301899] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:02 2019][2937584.308345] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:02 2019][2937584.314364] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:02 2019][2937584.320914] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.327709] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:02 2019][2937584.334155] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:02 2019][2937584.339828] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:02 2019][2937584.347407] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937584.352466] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.358742] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.365364] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.371636] osp-pre-12-2 S ffff8b5c0f89a080 0 117569 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.379000] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.381633] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:02 2019][2937584.387039] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.392194] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:02 2019][2937584.399165] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937584.404921] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:02 2019][2937584.411626] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937584.416687] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.422964] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.429599] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.435865] osp-syn-13-2 S ffff8b5c0f89e180 0 117570 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.443222] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.445851] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:02 2019][2937584.451954] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.457107] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:02 2019][2937584.464417] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937584.470196] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.477516] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:02 2019][2937584.484064] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.491312] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.498556] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.505890] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.513654] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.520904] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.528699] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.536379] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:02 2019][2937584.542837] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:02 2019][2937584.548857] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:02 2019][2937584.555415] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.562241] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:02 2019][2937584.568709] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:02 2019][2937584.574389] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:02 2019][2937584.581978] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937584.587055] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.593331] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.599949] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.606223] osp-pre-13-2 S ffff8b5c0f89b0c0 0 117571 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.613586] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.616219] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.621371] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:02 2019][2937584.628343] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937584.634129] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:02 2019][2937584.640869] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937584.645939] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.652234] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.658886] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.665160] osp-syn-39-2 S ffff8b5c0f89c100 0 117572 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.672541] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.675191] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:02 2019][2937584.681301] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.686453] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:02 2019][2937584.693770] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:02 2019][2937584.699558] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.706881] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:02 2019][2937584.713448] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.720721] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.727988] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.735332] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.743111] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.750359] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.758122] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.765784] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:02 2019][2937584.772228] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:02 2019][2937584.778254] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:02 2019][2937584.784833] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:02 2019][2937584.791652] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:02 2019][2937584.798103] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:02 2019][2937584.803782] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:02 2019][2937584.811354] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:02 2019][2937584.816438] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.822717] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:02 2019][2937584.829347] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:02 2019][2937584.835627] osp-pre-39-2 S ffff8b5c0f899040 0 117573 2 0x00000080 [Fri Apr 12 11:11:02 2019][2937584.842989] Call Trace: [Fri Apr 12 11:11:02 2019][2937584.845622] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:02 2019][2937584.851029] [] schedule+0x29/0x70 [Fri Apr 12 11:11:02 2019][2937584.856184] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:02 2019][2937584.863195] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937584.868977] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:03 2019][2937584.875714] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937584.880794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937584.887093] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937584.893723] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937584.899996] osp-syn-46-2 S ffff8b4f5b6930c0 0 117574 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937584.907357] Call Trace: [Fri Apr 12 11:11:03 2019][2937584.909990] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:03 2019][2937584.916091] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937584.921244] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:03 2019][2937584.928583] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937584.934386] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937584.941738] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:03 2019][2937584.948302] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:03 2019][2937584.955551] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:03 2019][2937584.962801] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937584.970134] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:03 2019][2937584.977896] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:03 2019][2937584.985144] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:03 2019][2937584.992906] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.000569] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:03 2019][2937585.007015] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:03 2019][2937585.013035] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:03 2019][2937585.019585] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.026381] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:03 2019][2937585.032825] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:03 2019][2937585.038495] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:03 2019][2937585.046068] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937585.051128] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.057404] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937585.064024] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.070297] osp-pre-46-2 S ffff8b4f5b694100 0 117575 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937585.077652] Call Trace: [Fri Apr 12 11:11:03 2019][2937585.080287] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:03 2019][2937585.085692] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937585.090845] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:03 2019][2937585.097809] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937585.103567] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:03 2019][2937585.110272] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937585.115334] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.121607] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937585.128228] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.134502] osp-syn-10-2 S ffff8b4f5b696180 0 117576 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937585.141866] Call Trace: [Fri Apr 12 11:11:03 2019][2937585.144547] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:03 2019][2937585.152124] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937585.157276] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:03 2019][2937585.164588] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937585.170364] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.177680] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:03 2019][2937585.184234] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.191484] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.198728] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.206061] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.213827] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.221072] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.228835] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.236499] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:03 2019][2937585.242945] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:03 2019][2937585.248962] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:03 2019][2937585.255513] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.262311] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:03 2019][2937585.268755] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:03 2019][2937585.274424] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:03 2019][2937585.282015] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937585.287076] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.293351] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937585.299972] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.306246] osp-pre-10-2 S ffff8b4f5b692080 0 117577 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937585.313609] Call Trace: [Fri Apr 12 11:11:03 2019][2937585.316240] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:03 2019][2937585.321647] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937585.326800] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:03 2019][2937585.333766] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937585.339538] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:03 2019][2937585.346244] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937585.351305] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.357579] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937585.364200] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.370475] osp-syn-11-2 S ffff8b4f5b695140 0 117578 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937585.377838] Call Trace: [Fri Apr 12 11:11:03 2019][2937585.380513] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:03 2019][2937585.388086] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937585.393239] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:03 2019][2937585.400564] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937585.406333] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.413652] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:03 2019][2937585.420210] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.427456] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.434701] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.442032] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.449797] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.457066] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.464832] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.472495] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:03 2019][2937585.478942] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:03 2019][2937585.484964] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:03 2019][2937585.491513] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.498307] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:03 2019][2937585.504755] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:03 2019][2937585.510423] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:03 2019][2937585.517994] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937585.523057] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.529332] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937585.535952] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.542227] osp-pre-11-2 S ffff8b4f5b690000 0 117579 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937585.549592] Call Trace: [Fri Apr 12 11:11:03 2019][2937585.552221] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:03 2019][2937585.557628] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937585.562781] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:03 2019][2937585.569746] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937585.575504] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:03 2019][2937585.582209] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937585.587268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.593545] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937585.600166] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.606440] osp-syn-14-2 S ffff8b5cfc205140 0 117580 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937585.613802] Call Trace: [Fri Apr 12 11:11:03 2019][2937585.616477] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:03 2019][2937585.624053] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937585.629204] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:03 2019][2937585.636517] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937585.642288] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.649605] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:03 2019][2937585.656155] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.663403] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.670647] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.677981] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.685743] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.692991] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.700756] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.708414] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:03 2019][2937585.714866] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:03 2019][2937585.720881] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:03 2019][2937585.727446] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:03 2019][2937585.734248] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:03 2019][2937585.740690] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:03 2019][2937585.746362] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:03 2019][2937585.753934] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937585.758993] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.765271] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937585.771891] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.778163] osp-pre-14-2 S ffff8b5cfc206180 0 117581 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937585.785520] Call Trace: [Fri Apr 12 11:11:03 2019][2937585.788152] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:03 2019][2937585.793558] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937585.798712] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:03 2019][2937585.805674] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:03 2019][2937585.811434] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:03 2019][2937585.818138] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:03 2019][2937585.823198] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.829475] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:03 2019][2937585.836094] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:03 2019][2937585.842368] osp-syn-26-2 S ffff8b5cfbc7d140 0 117582 2 0x00000080 [Fri Apr 12 11:11:03 2019][2937585.849740] Call Trace: [Fri Apr 12 11:11:03 2019][2937585.852415] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:03 2019][2937585.859992] [] schedule+0x29/0x70 [Fri Apr 12 11:11:03 2019][2937585.865143] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:04 2019][2937585.872453] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937585.878228] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.885546] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937585.892094] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.899342] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.906586] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.913935] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.921701] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.928947] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.936711] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.944371] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:04 2019][2937585.950816] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:04 2019][2937585.956828] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937585.963379] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:04 2019][2937585.970177] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:04 2019][2937585.976629] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:04 2019][2937585.982298] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:04 2019][2937585.989870] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:04 2019][2937585.994932] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.001206] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:04 2019][2937586.007828] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.014102] osp-pre-26-2 S ffff8b5cfbc7c100 0 117583 2 0x00000080 [Fri Apr 12 11:11:04 2019][2937586.021466] Call Trace: [Fri Apr 12 11:11:04 2019][2937586.024097] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:04 2019][2937586.029514] [] schedule+0x29/0x70 [Fri Apr 12 11:11:04 2019][2937586.034666] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:04 2019][2937586.041630] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937586.047388] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:04 2019][2937586.054091] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:04 2019][2937586.059145] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.065420] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:04 2019][2937586.072041] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.078315] osp-syn-31-2 S ffff8b5cfbc7a080 0 117584 2 0x00000080 [Fri Apr 12 11:11:04 2019][2937586.085678] Call Trace: [Fri Apr 12 11:11:04 2019][2937586.088356] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:04 2019][2937586.095929] [] schedule+0x29/0x70 [Fri Apr 12 11:11:04 2019][2937586.101081] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:04 2019][2937586.108390] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937586.114165] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.121483] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937586.128032] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.135279] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.142523] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.149855] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.157620] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.164868] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.172632] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.180292] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:04 2019][2937586.186739] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:04 2019][2937586.192760] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937586.199318] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.206116] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:04 2019][2937586.212570] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:04 2019][2937586.218240] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:04 2019][2937586.225811] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:04 2019][2937586.230870] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.237162] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:04 2019][2937586.243783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.250057] osp-pre-31-2 S ffff8b5cfbc7b0c0 0 117585 2 0x00000080 [Fri Apr 12 11:11:04 2019][2937586.257412] Call Trace: [Fri Apr 12 11:11:04 2019][2937586.260042] [] schedule+0x29/0x70 [Fri Apr 12 11:11:04 2019][2937586.265187] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:04 2019][2937586.272151] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937586.277911] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:04 2019][2937586.284614] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:04 2019][2937586.289675] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.295964] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:04 2019][2937586.302581] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.308852] osp-syn-42-2 S ffff8b5cfbc78000 0 117586 2 0x00000080 [Fri Apr 12 11:11:04 2019][2937586.316208] Call Trace: [Fri Apr 12 11:11:04 2019][2937586.318846] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:04 2019][2937586.325643] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:04 2019][2937586.331745] [] schedule+0x29/0x70 [Fri Apr 12 11:11:04 2019][2937586.336895] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:04 2019][2937586.344208] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937586.349981] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.357299] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937586.363849] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.371095] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.378339] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.385671] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.393436] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.400682] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.408447] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.416109] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:04 2019][2937586.422556] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:04 2019][2937586.428575] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937586.435125] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.441923] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:04 2019][2937586.448369] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:04 2019][2937586.454038] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:04 2019][2937586.461607] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:04 2019][2937586.466671] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.472943] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:04 2019][2937586.479566] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.485841] osp-pre-42-2 S ffff8b5cfbc7e180 0 117587 2 0x00000080 [Fri Apr 12 11:11:04 2019][2937586.493204] Call Trace: [Fri Apr 12 11:11:04 2019][2937586.495836] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:04 2019][2937586.501241] [] schedule+0x29/0x70 [Fri Apr 12 11:11:04 2019][2937586.506397] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:04 2019][2937586.513358] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937586.519117] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:04 2019][2937586.525820] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:04 2019][2937586.530882] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.537159] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:04 2019][2937586.543780] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.550055] osp-syn-9-2 S ffff8b5ceea6e180 0 117588 2 0x00000080 [Fri Apr 12 11:11:04 2019][2937586.557417] Call Trace: [Fri Apr 12 11:11:04 2019][2937586.560092] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:04 2019][2937586.567667] [] schedule+0x29/0x70 [Fri Apr 12 11:11:04 2019][2937586.572818] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:04 2019][2937586.580128] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937586.585901] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.593220] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937586.599768] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.607015] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.614261] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.621592] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.629359] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.636603] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.644370] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.652029] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:04 2019][2937586.658476] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:04 2019][2937586.664497] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937586.671047] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.677846] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:04 2019][2937586.684294] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:04 2019][2937586.689966] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:04 2019][2937586.697539] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:04 2019][2937586.702599] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.708874] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:04 2019][2937586.715495] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.721770] osp-pre-9-2 S ffff8b5ceea6d140 0 117589 2 0x00000080 [Fri Apr 12 11:11:04 2019][2937586.729133] Call Trace: [Fri Apr 12 11:11:04 2019][2937586.731764] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:04 2019][2937586.737171] [] schedule+0x29/0x70 [Fri Apr 12 11:11:04 2019][2937586.742339] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:04 2019][2937586.749304] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937586.755064] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:04 2019][2937586.761766] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:04 2019][2937586.766829] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.773105] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:04 2019][2937586.779725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:04 2019][2937586.786000] osp-syn-2-2 S ffff8b5ceea6a080 0 117590 2 0x00000080 [Fri Apr 12 11:11:04 2019][2937586.793364] Call Trace: [Fri Apr 12 11:11:04 2019][2937586.796003] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Fri Apr 12 11:11:04 2019][2937586.803404] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:04 2019][2937586.809505] [] schedule+0x29/0x70 [Fri Apr 12 11:11:04 2019][2937586.814657] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:04 2019][2937586.821969] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:04 2019][2937586.827742] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.835059] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:04 2019][2937586.841609] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.848856] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.856099] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.863434] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:04 2019][2937586.871213] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937586.878461] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937586.886227] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937586.893889] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:05 2019][2937586.900335] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:05 2019][2937586.906353] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:05 2019][2937586.912903] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:05 2019][2937586.919700] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:05 2019][2937586.926144] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:05 2019][2937586.931815] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:05 2019][2937586.939401] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:05 2019][2937586.944457] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937586.950733] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:05 2019][2937586.957352] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937586.963625] osp-pre-2-2 S ffff8b5ceea6c100 0 117591 2 0x00000080 [Fri Apr 12 11:11:05 2019][2937586.970982] Call Trace: [Fri Apr 12 11:11:05 2019][2937586.973611] [] schedule+0x29/0x70 [Fri Apr 12 11:11:05 2019][2937586.978765] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:05 2019][2937586.985729] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:05 2019][2937586.991489] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:05 2019][2937586.998191] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:05 2019][2937587.003254] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.009526] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:05 2019][2937587.016150] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.022424] osp-syn-43-2 S ffff8b4e2951b0c0 0 117592 2 0x00000080 [Fri Apr 12 11:11:05 2019][2937587.029787] Call Trace: [Fri Apr 12 11:11:05 2019][2937587.032425] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:05 2019][2937587.039221] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:05 2019][2937587.045321] [] schedule+0x29/0x70 [Fri Apr 12 11:11:05 2019][2937587.050476] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:05 2019][2937587.057789] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:05 2019][2937587.063559] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.070877] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:05 2019][2937587.077426] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.084671] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.091917] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.099236] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:05 2019][2937587.106217] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.113983] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.121227] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.129010] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.136673] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:05 2019][2937587.143118] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:05 2019][2937587.149139] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:05 2019][2937587.155688] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.162485] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:05 2019][2937587.168930] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:05 2019][2937587.174602] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:05 2019][2937587.182170] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:05 2019][2937587.187233] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.193508] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:05 2019][2937587.200130] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.206403] osp-pre-43-2 S ffff8b4e29518000 0 117593 2 0x00000080 [Fri Apr 12 11:11:05 2019][2937587.213767] Call Trace: [Fri Apr 12 11:11:05 2019][2937587.216399] [] schedule+0x29/0x70 [Fri Apr 12 11:11:05 2019][2937587.221550] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:05 2019][2937587.228512] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:11:05 2019][2937587.234008] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:05 2019][2937587.239768] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:05 2019][2937587.246474] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:05 2019][2937587.251532] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.257808] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:05 2019][2937587.264430] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.270704] osp-syn-19-2 S ffff8b5cf5d930c0 0 117594 2 0x00000080 [Fri Apr 12 11:11:05 2019][2937587.278065] Call Trace: [Fri Apr 12 11:11:05 2019][2937587.280744] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:05 2019][2937587.288315] [] schedule+0x29/0x70 [Fri Apr 12 11:11:05 2019][2937587.293468] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:05 2019][2937587.300779] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:05 2019][2937587.306550] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.313871] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:05 2019][2937587.320428] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.327676] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.334921] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.342240] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:05 2019][2937587.349229] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.356993] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.364237] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.372004] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.379664] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:05 2019][2937587.386112] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:05 2019][2937587.392133] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:05 2019][2937587.398682] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.405478] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:05 2019][2937587.411922] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:05 2019][2937587.417593] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:05 2019][2937587.425165] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:05 2019][2937587.430227] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.436502] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:05 2019][2937587.443122] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.449397] osp-pre-19-2 S ffff8b5cf5d94100 0 117595 2 0x00000080 [Fri Apr 12 11:11:05 2019][2937587.456759] Call Trace: [Fri Apr 12 11:11:05 2019][2937587.459391] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:05 2019][2937587.464802] [] schedule+0x29/0x70 [Fri Apr 12 11:11:05 2019][2937587.469950] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:05 2019][2937587.476914] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:05 2019][2937587.482674] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:05 2019][2937587.489407] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:05 2019][2937587.494465] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.500740] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:05 2019][2937587.507362] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.513636] osp-syn-18-2 S ffff8b5cf5d95140 0 117596 2 0x00000080 [Fri Apr 12 11:11:05 2019][2937587.520998] Call Trace: [Fri Apr 12 11:11:05 2019][2937587.523680] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:05 2019][2937587.531258] [] schedule+0x29/0x70 [Fri Apr 12 11:11:05 2019][2937587.536410] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:05 2019][2937587.543720] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:05 2019][2937587.549494] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.556811] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:05 2019][2937587.563359] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.570606] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.577850] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.585184] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.592952] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.600205] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.607967] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.615630] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:05 2019][2937587.622076] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:05 2019][2937587.628098] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:05 2019][2937587.634657] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.641466] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:05 2019][2937587.647912] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:05 2019][2937587.653584] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:05 2019][2937587.661156] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:05 2019][2937587.666215] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.672492] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:05 2019][2937587.679113] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.685387] osp-pre-18-2 S ffff8b5cf5d90000 0 117597 2 0x00000080 [Fri Apr 12 11:11:05 2019][2937587.692751] Call Trace: [Fri Apr 12 11:11:05 2019][2937587.695382] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:05 2019][2937587.700787] [] schedule+0x29/0x70 [Fri Apr 12 11:11:05 2019][2937587.705943] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:05 2019][2937587.712914] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:05 2019][2937587.718671] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:05 2019][2937587.725376] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:05 2019][2937587.730437] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.736712] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:05 2019][2937587.743333] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:05 2019][2937587.749606] osp-syn-25-2 S ffff8b5cf5d96180 0 117598 2 0x00000080 [Fri Apr 12 11:11:05 2019][2937587.756961] Call Trace: [Fri Apr 12 11:11:05 2019][2937587.759594] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:05 2019][2937587.765693] [] schedule+0x29/0x70 [Fri Apr 12 11:11:05 2019][2937587.770850] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:05 2019][2937587.778166] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:05 2019][2937587.783942] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.791259] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:05 2019][2937587.797810] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.805061] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.812311] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.819634] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:05 2019][2937587.826618] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.834382] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.841630] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.849395] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:05 2019][2937587.857053] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:05 2019][2937587.863501] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:05 2019][2937587.869521] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937587.876071] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:06 2019][2937587.882866] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:06 2019][2937587.889309] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:06 2019][2937587.894983] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:06 2019][2937587.902554] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937587.907615] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937587.913891] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937587.920510] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937587.926784] osp-pre-25-2 S ffff8b4e296da080 0 117599 2 0x00000080 [Fri Apr 12 11:11:06 2019][2937587.934147] Call Trace: [Fri Apr 12 11:11:06 2019][2937587.936781] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:06 2019][2937587.942187] [] schedule+0x29/0x70 [Fri Apr 12 11:11:06 2019][2937587.947343] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:06 2019][2937587.954314] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:06 2019][2937587.960110] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:06 2019][2937587.966818] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937587.971879] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937587.978155] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937587.984810] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937587.991085] osp-syn-17-2 S ffff8b4e295b30c0 0 117600 2 0x00000080 [Fri Apr 12 11:11:06 2019][2937587.998449] Call Trace: [Fri Apr 12 11:11:06 2019][2937588.001130] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:06 2019][2937588.008707] [] schedule+0x29/0x70 [Fri Apr 12 11:11:06 2019][2937588.013858] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:06 2019][2937588.021173] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:06 2019][2937588.026955] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.034278] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937588.040828] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.048070] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.055317] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.062650] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.070415] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.077662] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.085442] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.093104] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:06 2019][2937588.099550] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:06 2019][2937588.105569] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937588.112120] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.118920] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:06 2019][2937588.125391] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:06 2019][2937588.131060] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:06 2019][2937588.138641] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937588.143700] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.149975] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937588.156598] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.162871] osp-pre-17-2 S ffff8b56da615140 0 117601 2 0x00000080 [Fri Apr 12 11:11:06 2019][2937588.170236] Call Trace: [Fri Apr 12 11:11:06 2019][2937588.172867] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:06 2019][2937588.178275] [] schedule+0x29/0x70 [Fri Apr 12 11:11:06 2019][2937588.183436] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:06 2019][2937588.190407] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:06 2019][2937588.196167] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:06 2019][2937588.202878] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937588.207939] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.214213] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937588.220833] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.227109] osp-syn-0-2 S ffff8b56da610000 0 117602 2 0x00000080 [Fri Apr 12 11:11:06 2019][2937588.234472] Call Trace: [Fri Apr 12 11:11:06 2019][2937588.237114] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:06 2019][2937588.243907] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:06 2019][2937588.250009] [] schedule+0x29/0x70 [Fri Apr 12 11:11:06 2019][2937588.255161] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:06 2019][2937588.262500] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:06 2019][2937588.268274] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.275602] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937588.282156] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.289410] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.296664] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.304026] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.311785] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.319034] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.326837] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.334504] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:06 2019][2937588.340949] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:06 2019][2937588.346969] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937588.353519] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.360317] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:06 2019][2937588.366769] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:06 2019][2937588.372441] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:06 2019][2937588.380013] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937588.385074] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.391347] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937588.397969] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.404241] osp-pre-0-2 S ffff8b56da612080 0 117603 2 0x00000080 [Fri Apr 12 11:11:06 2019][2937588.411606] Call Trace: [Fri Apr 12 11:11:06 2019][2937588.414239] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:06 2019][2937588.419645] [] schedule+0x29/0x70 [Fri Apr 12 11:11:06 2019][2937588.424799] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:06 2019][2937588.431762] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:06 2019][2937588.437520] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:06 2019][2937588.444222] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937588.449284] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.455560] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937588.462180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.468455] osp-syn-5-2 S ffff8b56da614100 0 117604 2 0x00000080 [Fri Apr 12 11:11:06 2019][2937588.475818] Call Trace: [Fri Apr 12 11:11:06 2019][2937588.478499] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:06 2019][2937588.486076] [] schedule+0x29/0x70 [Fri Apr 12 11:11:06 2019][2937588.491229] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:06 2019][2937588.498539] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:06 2019][2937588.504316] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.511631] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937588.518178] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.525445] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.532689] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.540024] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.547795] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.555039] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.562811] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.570472] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:06 2019][2937588.576920] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:06 2019][2937588.582942] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937588.589490] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.596303] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:06 2019][2937588.602750] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:06 2019][2937588.608422] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:06 2019][2937588.615992] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937588.621052] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.627328] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937588.633948] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.640215] osp-pre-5-2 S ffff8b56da6130c0 0 117605 2 0x00000080 [Fri Apr 12 11:11:06 2019][2937588.647577] Call Trace: [Fri Apr 12 11:11:06 2019][2937588.650209] [] schedule+0x29/0x70 [Fri Apr 12 11:11:06 2019][2937588.655362] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:06 2019][2937588.662326] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:06 2019][2937588.668085] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:06 2019][2937588.674787] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937588.679849] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.686125] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937588.692745] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.699019] osp-syn-44-2 S ffff8b56da616180 0 117606 2 0x00000080 [Fri Apr 12 11:11:06 2019][2937588.706375] Call Trace: [Fri Apr 12 11:11:06 2019][2937588.709004] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:06 2019][2937588.715105] [] schedule+0x29/0x70 [Fri Apr 12 11:11:06 2019][2937588.720259] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:06 2019][2937588.727568] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:06 2019][2937588.733341] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.740661] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937588.747208] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.754455] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.761700] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.769035] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.776799] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.784046] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.791813] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.799478] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:06 2019][2937588.805927] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:06 2019][2937588.811946] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:06 2019][2937588.818498] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:06 2019][2937588.825292] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:06 2019][2937588.831739] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:06 2019][2937588.837407] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:06 2019][2937588.844980] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:06 2019][2937588.850054] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.856324] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:06 2019][2937588.862945] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:06 2019][2937588.869218] osp-pre-44-2 S ffff8b5cf6ba4100 0 117607 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937588.876573] Call Trace: [Fri Apr 12 11:11:07 2019][2937588.879206] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:07 2019][2937588.884612] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937588.889766] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:07 2019][2937588.896729] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937588.902489] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:07 2019][2937588.909191] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:07 2019][2937588.914252] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937588.920528] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:07 2019][2937588.927149] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937588.933422] osp-syn-32-2 S ffff8b5cf6ba30c0 0 117608 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937588.940778] Call Trace: [Fri Apr 12 11:11:07 2019][2937588.943408] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:07 2019][2937588.949510] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937588.954663] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:07 2019][2937588.961975] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937588.967762] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937588.975082] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:07 2019][2937588.981632] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:07 2019][2937588.988879] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:07 2019][2937588.996123] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.003456] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.011218] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.018464] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.026233] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.033892] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:07 2019][2937589.040339] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:07 2019][2937589.046358] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:07 2019][2937589.052910] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.059706] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:07 2019][2937589.066151] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:07 2019][2937589.071825] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:07 2019][2937589.079399] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:07 2019][2937589.084461] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.090737] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:07 2019][2937589.097359] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.103631] osp-pre-32-2 S ffff8b5cf6ba0000 0 117609 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937589.110995] Call Trace: [Fri Apr 12 11:11:07 2019][2937589.113627] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937589.118780] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:07 2019][2937589.125744] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937589.131501] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:07 2019][2937589.138204] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:07 2019][2937589.143265] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.149543] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:07 2019][2937589.156163] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.162436] osp-syn-21-2 S ffff8b5cf6ba1040 0 117610 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937589.169799] Call Trace: [Fri Apr 12 11:11:07 2019][2937589.172474] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:07 2019][2937589.180047] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937589.185203] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:07 2019][2937589.192512] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937589.198285] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.205603] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:07 2019][2937589.212152] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.219397] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.226644] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.233977] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.241741] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.248988] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.256753] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.264418] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:07 2019][2937589.270859] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:07 2019][2937589.276901] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:07 2019][2937589.283470] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.290292] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:07 2019][2937589.296744] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:07 2019][2937589.302414] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:07 2019][2937589.309992] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:07 2019][2937589.315054] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.321329] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:07 2019][2937589.327950] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.334225] osp-pre-21-2 S ffff8b5cf6ba6180 0 117611 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937589.341595] Call Trace: [Fri Apr 12 11:11:07 2019][2937589.344238] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937589.349400] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:07 2019][2937589.356387] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:11:07 2019][2937589.361881] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937589.367639] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:07 2019][2937589.374344] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:07 2019][2937589.379414] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.385690] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:07 2019][2937589.392355] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.398635] osp-syn-16-2 S ffff8b5cf3fc8000 0 117612 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937589.405999] Call Trace: [Fri Apr 12 11:11:07 2019][2937589.408686] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:07 2019][2937589.416267] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937589.421429] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:07 2019][2937589.428748] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937589.434533] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.441856] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:07 2019][2937589.448423] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.455675] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.462922] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.470257] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.478037] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.485282] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.493048] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.500720] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:07 2019][2937589.507170] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:07 2019][2937589.513194] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:07 2019][2937589.519750] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.526547] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:07 2019][2937589.533000] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:07 2019][2937589.538672] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:07 2019][2937589.546263] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:07 2019][2937589.551325] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.557599] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:07 2019][2937589.564218] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.570492] osp-pre-16-2 S ffff8b5cf3fcb0c0 0 117613 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937589.577853] Call Trace: [Fri Apr 12 11:11:07 2019][2937589.580485] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937589.585638] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:07 2019][2937589.592601] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:11:07 2019][2937589.598095] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937589.603858] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:07 2019][2937589.610593] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:07 2019][2937589.615656] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.621930] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:07 2019][2937589.628552] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.634824] osp-syn-41-2 S ffff8b5cf3fcd140 0 117614 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937589.642191] Call Trace: [Fri Apr 12 11:11:07 2019][2937589.644888] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:07 2019][2937589.652472] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937589.657628] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:07 2019][2937589.664946] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937589.670728] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.678045] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:07 2019][2937589.684605] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.691883] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.699127] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.706450] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:07 2019][2937589.713439] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.721213] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.728459] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.736232] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.743892] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:07 2019][2937589.750339] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:07 2019][2937589.756358] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:07 2019][2937589.762909] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:07 2019][2937589.769724] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:07 2019][2937589.776177] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:07 2019][2937589.781854] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:07 2019][2937589.789430] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:07 2019][2937589.794507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.800781] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:07 2019][2937589.807402] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:07 2019][2937589.813675] osp-pre-41-2 S ffff8b5cf3fce180 0 117615 2 0x00000080 [Fri Apr 12 11:11:07 2019][2937589.821037] Call Trace: [Fri Apr 12 11:11:07 2019][2937589.823668] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:07 2019][2937589.829076] [] schedule+0x29/0x70 [Fri Apr 12 11:11:07 2019][2937589.834230] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:07 2019][2937589.841204] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:07 2019][2937589.846960] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:08 2019][2937589.853664] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937589.858735] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937589.865013] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937589.871631] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937589.877903] osp-syn-4-2 S ffff8b5cf3fca080 0 117616 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937589.885276] Call Trace: [Fri Apr 12 11:11:08 2019][2937589.887920] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:08 2019][2937589.894712] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:08 2019][2937589.900815] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937589.905984] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:08 2019][2937589.913302] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:08 2019][2937589.919076] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937589.926392] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:08 2019][2937589.932958] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:08 2019][2937589.940213] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:08 2019][2937589.947460] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937589.954778] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:08 2019][2937589.961767] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:08 2019][2937589.969534] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:08 2019][2937589.976778] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:08 2019][2937589.984545] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937589.992205] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:08 2019][2937589.998674] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:08 2019][2937590.004692] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:08 2019][2937590.011249] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.018045] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:08 2019][2937590.024489] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:08 2019][2937590.030162] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:08 2019][2937590.037744] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937590.042813] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.049086] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937590.055725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.061996] osp-pre-4-2 S ffff8b5cee352080 0 117617 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937590.069360] Call Trace: [Fri Apr 12 11:11:08 2019][2937590.071993] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:08 2019][2937590.077401] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937590.082553] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:08 2019][2937590.089515] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:08 2019][2937590.095275] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:08 2019][2937590.101978] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937590.107049] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.113325] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937590.119956] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.126230] osp-syn-23-2 S ffff8b5cee351040 0 117618 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937590.133591] Call Trace: [Fri Apr 12 11:11:08 2019][2937590.136238] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:08 2019][2937590.142340] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937590.147491] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:08 2019][2937590.154803] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:08 2019][2937590.160582] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.167903] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:08 2019][2937590.174474] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.181743] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.188986] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.196327] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.204097] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.211337] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.219101] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.226764] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:08 2019][2937590.233213] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:08 2019][2937590.239230] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:08 2019][2937590.245780] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.252580] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:08 2019][2937590.259035] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:08 2019][2937590.264720] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:08 2019][2937590.272293] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937590.277353] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.283630] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937590.290260] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.296531] osp-pre-23-2 S ffff8b5c066ed140 0 117619 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937590.303894] Call Trace: [Fri Apr 12 11:11:08 2019][2937590.306536] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:08 2019][2937590.311953] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937590.317119] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:08 2019][2937590.324107] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:08 2019][2937590.329868] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:08 2019][2937590.336574] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937590.341634] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.347911] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937590.354530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.360805] osp-syn-33-2 S ffff8b5c066ee180 0 117620 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937590.368186] Call Trace: [Fri Apr 12 11:11:08 2019][2937590.370839] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:08 2019][2937590.377639] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:08 2019][2937590.383760] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937590.388909] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:08 2019][2937590.396230] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:08 2019][2937590.402005] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.409320] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:08 2019][2937590.415878] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.423123] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.430370] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.437700] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.445467] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.452711] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.460484] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.468162] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:08 2019][2937590.474611] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:08 2019][2937590.480641] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:08 2019][2937590.487216] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.494045] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:08 2019][2937590.500501] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:08 2019][2937590.506182] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:08 2019][2937590.513769] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937590.518832] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.525115] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937590.531735] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.538006] osp-pre-33-2 S ffff8b5c066e8000 0 117621 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937590.545364] Call Trace: [Fri Apr 12 11:11:08 2019][2937590.547993] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937590.553149] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:08 2019][2937590.560119] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:08 2019][2937590.565899] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:08 2019][2937590.572607] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937590.577688] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.583969] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937590.590599] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.596873] osp-syn-20-2 S ffff8b5c066ec100 0 117622 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937590.604238] Call Trace: [Fri Apr 12 11:11:08 2019][2937590.606878] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:08 2019][2937590.613672] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:08 2019][2937590.619774] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937590.624936] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:08 2019][2937590.632256] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:08 2019][2937590.638031] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.645366] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:08 2019][2937590.651929] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.659172] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.666416] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.673752] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.681516] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.688763] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.696563] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.704234] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:08 2019][2937590.710681] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:08 2019][2937590.716701] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:08 2019][2937590.723257] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:08 2019][2937590.730055] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:08 2019][2937590.736498] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:08 2019][2937590.742170] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:08 2019][2937590.749740] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937590.754803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.761078] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937590.767699] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.773971] osp-pre-20-2 S ffff8b5c066eb0c0 0 117623 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937590.781335] Call Trace: [Fri Apr 12 11:11:08 2019][2937590.783969] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937590.789118] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:08 2019][2937590.796081] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:08 2019][2937590.801840] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:08 2019][2937590.808548] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:08 2019][2937590.813617] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.819908] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:08 2019][2937590.826530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:08 2019][2937590.832803] osp-syn-7-2 S ffff8b5c066ea080 0 117624 2 0x00000080 [Fri Apr 12 11:11:08 2019][2937590.840167] Call Trace: [Fri Apr 12 11:11:08 2019][2937590.842798] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:08 2019][2937590.848900] [] schedule+0x29/0x70 [Fri Apr 12 11:11:08 2019][2937590.854053] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:09 2019][2937590.861370] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:09 2019][2937590.867149] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.874480] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:09 2019][2937590.881042] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.888308] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.895575] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.902913] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.910677] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.917923] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.925686] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.933369] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:09 2019][2937590.939824] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:09 2019][2937590.945846] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:09 2019][2937590.952401] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:09 2019][2937590.959196] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:09 2019][2937590.965639] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:09 2019][2937590.971316] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:09 2019][2937590.978896] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:09 2019][2937590.983955] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937590.990234] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:09 2019][2937590.996867] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.003141] osp-pre-7-2 S ffff8b5e5e4ae180 0 117625 2 0x00000080 [Fri Apr 12 11:11:09 2019][2937591.010512] Call Trace: [Fri Apr 12 11:11:09 2019][2937591.013156] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:09 2019][2937591.018561] [] schedule+0x29/0x70 [Fri Apr 12 11:11:09 2019][2937591.023715] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:09 2019][2937591.030698] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:09 2019][2937591.036462] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:09 2019][2937591.043168] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:09 2019][2937591.048237] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.054529] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:09 2019][2937591.061149] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.067423] osp-syn-40-2 S ffff8b6cf91930c0 0 117626 2 0x00000080 [Fri Apr 12 11:11:09 2019][2937591.074795] Call Trace: [Fri Apr 12 11:11:09 2019][2937591.077437] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:09 2019][2937591.083537] [] schedule+0x29/0x70 [Fri Apr 12 11:11:09 2019][2937591.088701] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:09 2019][2937591.096024] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:09 2019][2937591.101806] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.109137] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:09 2019][2937591.115692] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.122938] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.130184] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.137515] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.145281] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.152527] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.160291] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.167953] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:09 2019][2937591.174401] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:09 2019][2937591.180418] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:09 2019][2937591.186968] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.193765] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:09 2019][2937591.200209] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:09 2019][2937591.205882] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:09 2019][2937591.213452] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:09 2019][2937591.218513] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.224788] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:09 2019][2937591.231408] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.237683] osp-pre-40-2 S ffff8b6cf9190000 0 117627 2 0x00000080 [Fri Apr 12 11:11:09 2019][2937591.245063] Call Trace: [Fri Apr 12 11:11:09 2019][2937591.247696] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:09 2019][2937591.253101] [] schedule+0x29/0x70 [Fri Apr 12 11:11:09 2019][2937591.258258] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:09 2019][2937591.265227] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:09 2019][2937591.270985] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:09 2019][2937591.277688] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:09 2019][2937591.282740] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.289010] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:09 2019][2937591.295630] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.301903] osp-syn-45-2 S ffff8b6cf9196180 0 117628 2 0x00000080 [Fri Apr 12 11:11:09 2019][2937591.309259] Call Trace: [Fri Apr 12 11:11:09 2019][2937591.311938] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:09 2019][2937591.319508] [] schedule+0x29/0x70 [Fri Apr 12 11:11:09 2019][2937591.324660] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:09 2019][2937591.331970] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:09 2019][2937591.337746] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.345061] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:09 2019][2937591.351613] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.358857] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.366101] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.373436] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.381200] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.388447] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.396212] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.403875] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:09 2019][2937591.410318] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:09 2019][2937591.416338] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:09 2019][2937591.422890] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.429685] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:09 2019][2937591.436130] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:09 2019][2937591.441801] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:09 2019][2937591.449372] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:09 2019][2937591.454431] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.460718] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:09 2019][2937591.467338] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.473611] osp-pre-45-2 S ffff8b6cf9194100 0 117629 2 0x00000080 [Fri Apr 12 11:11:09 2019][2937591.480966] Call Trace: [Fri Apr 12 11:11:09 2019][2937591.483597] [] schedule+0x29/0x70 [Fri Apr 12 11:11:09 2019][2937591.488752] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:09 2019][2937591.495714] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:09 2019][2937591.501475] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:09 2019][2937591.508175] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:09 2019][2937591.513237] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.519527] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:09 2019][2937591.526143] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.532415] osp-syn-6-2 S ffff8b6cf9192080 0 117630 2 0x00000080 [Fri Apr 12 11:11:09 2019][2937591.539773] Call Trace: [Fri Apr 12 11:11:09 2019][2937591.542402] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:09 2019][2937591.548503] [] schedule+0x29/0x70 [Fri Apr 12 11:11:09 2019][2937591.553656] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:09 2019][2937591.560968] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:09 2019][2937591.566744] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.574060] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:09 2019][2937591.580616] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.587862] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.595109] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.602442] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.610205] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.617449] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.625216] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.632881] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:09 2019][2937591.639322] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:09 2019][2937591.645342] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:09 2019][2937591.651898] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:09 2019][2937591.658696] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:09 2019][2937591.665143] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:09 2019][2937591.670813] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:09 2019][2937591.678386] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:09 2019][2937591.683445] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.689713] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:09 2019][2937591.696332] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.702599] osp-pre-6-2 S ffff8b6cf9195140 0 117631 2 0x00000080 [Fri Apr 12 11:11:09 2019][2937591.709963] Call Trace: [Fri Apr 12 11:11:09 2019][2937591.712594] [] schedule+0x29/0x70 [Fri Apr 12 11:11:09 2019][2937591.717747] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:09 2019][2937591.724712] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:09 2019][2937591.730469] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:09 2019][2937591.737171] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:09 2019][2937591.742232] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.748500] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:09 2019][2937591.755120] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:09 2019][2937591.761395] osp-syn-24-2 S ffff8b69ea63a080 [2937591.764893] LNet: Service thread pid 118588 was inactive for 200.71s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 11:11:09 2019][2937591.764896] Pid: 118588, comm: mdt02_109 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:11:09 2019][2937591.764896] Call Trace: [Fri Apr 12 11:11:09 2019][2937591.764909] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:09 2019][2937591.764959] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:09 2019][2937591.764999] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:09 2019][2937591.765039] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:09 2019][2937591.765051] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:09 2019][2937591.765065] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:09 2019][2937591.765077] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:09 2019][2937591.765099] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:09 2019][2937591.765113] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:10 2019][2937591.765129] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:10 2019][2937591.765143] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:10 2019][2937591.765159] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:10 2019][2937591.765172] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:10 2019][2937591.765185] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:10 2019][2937591.765244] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:10 2019][2937591.765294] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:10 2019][2937591.765337] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937591.765341] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937591.765345] [] 0xffffffffffffffff [Fri Apr 12 11:11:10 2019][2937591.765380] LustreError: dumping log to /tmp/lustre-log.1555092669.118588 [Fri Apr 12 11:11:10 2019] [Fri Apr 12 11:11:10 2019][2937591.927677] 0 117632 2 0x00000080 [Fri Apr 12 11:11:10 2019][2937591.932149] Call Trace: [Fri Apr 12 11:11:10 2019][2937591.934820] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:10 2019][2937591.941619] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:10 2019][2937591.947730] [] schedule+0x29/0x70 [Fri Apr 12 11:11:10 2019][2937591.952886] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:10 2019][2937591.960216] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:10 2019][2937591.965992] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937591.973320] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:10 2019][2937591.979912] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:10 2019][2937591.987181] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:10 2019][2937591.994424] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.001771] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.009550] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.016799] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.024568] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.032241] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:10 2019][2937592.038705] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:10 2019][2937592.044725] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:10 2019][2937592.051327] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.058132] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:10 2019][2937592.064609] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:10 2019][2937592.070280] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:10 2019][2937592.077876] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937592.083001] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.089289] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937592.095911] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.102185] osp-pre-24-2 S ffff8b69ea63c100 0 117633 2 0x00000080 [Fri Apr 12 11:11:10 2019][2937592.109559] Call Trace: [Fri Apr 12 11:11:10 2019][2937592.112187] [] schedule+0x29/0x70 [Fri Apr 12 11:11:10 2019][2937592.117341] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:10 2019][2937592.124323] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:10 2019][2937592.130097] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:10 2019][2937592.136809] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937592.141890] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.148163] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937592.154786] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.161067] osp-syn-47-2 S ffff8b69ea639040 0 117634 2 0x00000080 [Fri Apr 12 11:11:10 2019][2937592.168432] Call Trace: [Fri Apr 12 11:11:10 2019][2937592.171064] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:10 2019][2937592.177164] [] schedule+0x29/0x70 [Fri Apr 12 11:11:10 2019][2937592.182352] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:10 2019][2937592.189705] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:10 2019][2937592.195481] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.202822] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:10 2019][2937592.209372] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.216619] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.223895] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.231247] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.239027] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.246400] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.254180] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.261840] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:10 2019][2937592.268288] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:10 2019][2937592.274306] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:10 2019][2937592.280873] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.287670] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:10 2019][2937592.294117] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:10 2019][2937592.299786] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:10 2019][2937592.307377] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937592.312436] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.318710] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937592.325332] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.331606] osp-pre-47-2 S ffff8b69ea638000 0 117635 2 0x00000080 [Fri Apr 12 11:11:10 2019][2937592.338969] Call Trace: [Fri Apr 12 11:11:10 2019][2937592.341600] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:10 2019][2937592.347008] [] schedule+0x29/0x70 [Fri Apr 12 11:11:10 2019][2937592.352161] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:10 2019][2937592.359123] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:10 2019][2937592.364900] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:10 2019][2937592.371602] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937592.376655] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.382925] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937592.389545] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.395836] osp-syn-38-2 S ffff8b69ea63d140 0 117636 2 0x00000080 [Fri Apr 12 11:11:10 2019][2937592.403199] Call Trace: [Fri Apr 12 11:11:10 2019][2937592.405831] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:10 2019][2937592.411939] [] schedule+0x29/0x70 [Fri Apr 12 11:11:10 2019][2937592.417103] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:10 2019][2937592.424412] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:10 2019][2937592.430208] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.437532] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:10 2019][2937592.444087] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.451336] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.458577] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.465915] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.473684] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.480947] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.488713] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.496372] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:10 2019][2937592.502819] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:10 2019][2937592.508839] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:10 2019][2937592.515408] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.522205] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:10 2019][2937592.528668] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:10 2019][2937592.534337] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:10 2019][2937592.541918] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937592.546979] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.553264] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937592.559894] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.566183] osp-pre-38-2 S ffff8b6cf90d5140 0 117637 2 0x00000080 [Fri Apr 12 11:11:10 2019][2937592.573583] Call Trace: [Fri Apr 12 11:11:10 2019][2937592.576211] [] schedule+0x29/0x70 [Fri Apr 12 11:11:10 2019][2937592.581367] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:10 2019][2937592.588347] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:10 2019][2937592.594105] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:10 2019][2937592.600822] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937592.605916] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.612190] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937592.618807] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.625121] osp-syn-15-2 S ffff8b6cf90d6180 0 117638 2 0x00000080 [Fri Apr 12 11:11:10 2019][2937592.632498] Call Trace: [Fri Apr 12 11:11:10 2019][2937592.635181] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:10 2019][2937592.642782] [] schedule+0x29/0x70 [Fri Apr 12 11:11:10 2019][2937592.647935] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:10 2019][2937592.655255] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:10 2019][2937592.661039] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.668355] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:10 2019][2937592.674915] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.682167] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.689422] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.696770] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.704537] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.711808] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.719571] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.727234] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:10 2019][2937592.733679] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:10 2019][2937592.739700] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:10 2019][2937592.746248] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:10 2019][2937592.753055] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:10 2019][2937592.759499] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:10 2019][2937592.765176] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:10 2019][2937592.772764] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937592.777821] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.784098] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937592.790720] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.796993] osp-pre-15-2 S ffff8b6cf90d30c0 0 117639 2 0x00000080 [Fri Apr 12 11:11:10 2019][2937592.804363] Call Trace: [Fri Apr 12 11:11:10 2019][2937592.806997] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:10 2019][2937592.812404] [] schedule+0x29/0x70 [Fri Apr 12 11:11:10 2019][2937592.817557] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:10 2019][2937592.824531] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:10 2019][2937592.830289] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:10 2019][2937592.837000] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:10 2019][2937592.842059] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:10 2019][2937592.848335] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:10 2019][2937592.854957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937592.861230] osp-syn-29-2 S ffff8b6cf90d0000 0 117640 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937592.868593] Call Trace: [Fri Apr 12 11:11:11 2019][2937592.871237] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:11 2019][2937592.878029] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:11 2019][2937592.884129] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937592.889283] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:11 2019][2937592.896594] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937592.902369] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937592.909686] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:11 2019][2937592.916243] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:11 2019][2937592.923495] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:11 2019][2937592.930744] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937592.938062] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:11 2019][2937592.945054] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:11 2019][2937592.952821] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:11 2019][2937592.960070] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:11 2019][2937592.967836] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937592.975507] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:11 2019][2937592.981971] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:11 2019][2937592.987991] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:11 2019][2937592.994550] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.001354] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:11 2019][2937593.007834] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:11 2019][2937593.013505] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:11 2019][2937593.021083] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:11 2019][2937593.026143] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.032418] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:11 2019][2937593.039041] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.045316] osp-pre-29-2 S ffff8b6cf90d1040 0 117641 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937593.052696] Call Trace: [Fri Apr 12 11:11:11 2019][2937593.055325] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937593.060488] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:11 2019][2937593.067462] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937593.073220] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:11 2019][2937593.079920] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:11 2019][2937593.084985] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.091267] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:11 2019][2937593.097888] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.104162] osp-syn-1-2 S ffff8b6cf90d2080 0 117642 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937593.111534] Call Trace: [Fri Apr 12 11:11:11 2019][2937593.114175] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:11 2019][2937593.120971] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:11 2019][2937593.127069] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937593.132224] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:11 2019][2937593.139543] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937593.145317] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.152634] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:11 2019][2937593.159194] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.166444] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.173695] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.181011] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:11 2019][2937593.188002] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.195769] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.203020] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.210789] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.218457] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:11 2019][2937593.224901] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:11 2019][2937593.230919] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:11 2019][2937593.237471] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.244270] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:11 2019][2937593.250725] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:11 2019][2937593.256395] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:11 2019][2937593.264001] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:11 2019][2937593.269061] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.275335] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:11 2019][2937593.281957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.288229] osp-pre-1-2 S ffff8b6cfc58a080 0 117643 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937593.295591] Call Trace: [Fri Apr 12 11:11:11 2019][2937593.298222] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937593.303377] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:11 2019][2937593.310340] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937593.316108] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:11 2019][2937593.322837] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:11 2019][2937593.327899] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.334174] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:11 2019][2937593.340796] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.347070] osp-syn-22-2 S ffff8b6cfc58c100 0 117644 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937593.354441] Call Trace: [Fri Apr 12 11:11:11 2019][2937593.357072] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:11 2019][2937593.363174] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937593.368327] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:11 2019][2937593.375647] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937593.381424] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.388748] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:11 2019][2937593.395303] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.402551] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.409797] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.417137] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.424905] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.432152] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.439924] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.447586] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:11 2019][2937593.454063] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:11 2019][2937593.460087] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:11 2019][2937593.466650] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.473449] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:11 2019][2937593.479903] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:11 2019][2937593.485574] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:11 2019][2937593.493153] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:11 2019][2937593.498214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.504485] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:11 2019][2937593.511109] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.517407] osp-pre-22-2 S ffff8b6cfc58d140 0 117645 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937593.524769] Call Trace: [Fri Apr 12 11:11:11 2019][2937593.527404] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937593.532565] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:11 2019][2937593.539535] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937593.545297] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:11 2019][2937593.552015] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:11 2019][2937593.557077] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.563352] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:11 2019][2937593.569974] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.576246] osp-syn-3-2 S ffff8b6cfc589040 0 117646 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937593.583610] Call Trace: [Fri Apr 12 11:11:11 2019][2937593.586251] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:11 2019][2937593.593047] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:11 2019][2937593.599145] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937593.604299] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:11 2019][2937593.611612] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937593.617386] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.624702] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:11 2019][2937593.631259] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.638506] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.645758] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.653094] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.660866] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.668114] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.675887] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.683546] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:11 2019][2937593.689992] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:11 2019][2937593.696012] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:11 2019][2937593.702573] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.709403] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:11 2019][2937593.715855] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:11 2019][2937593.721528] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:11 2019][2937593.729116] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:11 2019][2937593.734176] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.740455] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:11 2019][2937593.747072] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.753363] osp-pre-3-2 S ffff8b6cfc58b0c0 0 117647 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937593.760733] Call Trace: [Fri Apr 12 11:11:11 2019][2937593.763375] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937593.768529] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:11 2019][2937593.775499] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937593.781258] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:11 2019][2937593.787962] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:11 2019][2937593.793024] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.799298] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:11 2019][2937593.805924] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:11 2019][2937593.812195] osp-syn-27-2 S ffff8b6cfc588000 0 117648 2 0x00000080 [Fri Apr 12 11:11:11 2019][2937593.819564] Call Trace: [Fri Apr 12 11:11:11 2019][2937593.822207] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:11 2019][2937593.829002] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:11 2019][2937593.835103] [] schedule+0x29/0x70 [Fri Apr 12 11:11:11 2019][2937593.840257] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:11 2019][2937593.847591] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:11 2019][2937593.853367] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:11 2019][2937593.860682] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937593.867246] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937593.874498] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:12 2019][2937593.881750] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937593.889084] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937593.896860] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937593.904112] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937593.911879] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937593.919544] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:12 2019][2937593.925992] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:12 2019][2937593.932011] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937593.938570] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:12 2019][2937593.945365] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:12 2019][2937593.951812] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:12 2019][2937593.957482] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:12 2019][2937593.965062] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:12 2019][2937593.970121] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937593.976399] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:12 2019][2937593.983018] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937593.989296] osp-pre-27-2 S ffff8b6448e36180 0 117649 2 0x00000080 [Fri Apr 12 11:11:12 2019][2937593.996675] Call Trace: [Fri Apr 12 11:11:12 2019][2937593.999316] [] schedule+0x29/0x70 [Fri Apr 12 11:11:12 2019][2937594.004468] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:12 2019][2937594.011439] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:11:12 2019][2937594.016935] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:12 2019][2937594.022719] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:12 2019][2937594.029430] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:12 2019][2937594.034490] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.040767] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:12 2019][2937594.047387] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.053661] osp-syn-37-2 S ffff8b6448e35140 0 117650 2 0x00000080 [Fri Apr 12 11:11:12 2019][2937594.061025] Call Trace: [Fri Apr 12 11:11:12 2019][2937594.063656] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:12 2019][2937594.069756] [] schedule+0x29/0x70 [Fri Apr 12 11:11:12 2019][2937594.074909] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:12 2019][2937594.082231] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:12 2019][2937594.088006] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.095328] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937594.101898] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.109191] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.116442] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.123813] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.131578] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.138824] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.146608] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.154269] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:12 2019][2937594.160718] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:12 2019][2937594.166752] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937594.173306] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.180108] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:12 2019][2937594.186552] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:12 2019][2937594.192224] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:12 2019][2937594.199792] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:12 2019][2937594.204856] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.211129] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:12 2019][2937594.217752] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.224025] osp-pre-37-2 S ffff8b6448e32080 0 117651 2 0x00000080 [Fri Apr 12 11:11:12 2019][2937594.231389] Call Trace: [Fri Apr 12 11:11:12 2019][2937594.234022] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:12 2019][2937594.239427] [] schedule+0x29/0x70 [Fri Apr 12 11:11:12 2019][2937594.244582] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:12 2019][2937594.251562] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:12 2019][2937594.257321] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:12 2019][2937594.264033] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:12 2019][2937594.269096] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.275370] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:12 2019][2937594.281992] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.288265] osp-syn-36-2 S ffff8b6448e34100 0 117652 2 0x00000080 [Fri Apr 12 11:11:12 2019][2937594.295626] Call Trace: [Fri Apr 12 11:11:12 2019][2937594.298258] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:12 2019][2937594.304360] [] schedule+0x29/0x70 [Fri Apr 12 11:11:12 2019][2937594.309513] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:12 2019][2937594.316823] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:12 2019][2937594.322601] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.329923] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937594.336495] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.343747] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.351001] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.358333] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.366112] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.373365] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.381136] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.388804] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:12 2019][2937594.395251] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:12 2019][2937594.401269] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937594.407832] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.414634] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:12 2019][2937594.421087] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:12 2019][2937594.426760] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:12 2019][2937594.434336] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:12 2019][2937594.439397] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.445675] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:12 2019][2937594.452294] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.458568] osp-pre-36-2 S ffff8b6448e30000 0 117653 2 0x00000080 [Fri Apr 12 11:11:12 2019][2937594.465934] Call Trace: [Fri Apr 12 11:11:12 2019][2937594.468572] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:12 2019][2937594.473982] [] schedule+0x29/0x70 [Fri Apr 12 11:11:12 2019][2937594.479133] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:12 2019][2937594.486107] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:12 2019][2937594.491863] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:12 2019][2937594.498568] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:12 2019][2937594.503628] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.509903] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:12 2019][2937594.516543] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.522816] osp-syn-8-2 S ffff8b5da99ce180 0 117654 2 0x00000080 [Fri Apr 12 11:11:12 2019][2937594.530187] Call Trace: [Fri Apr 12 11:11:12 2019][2937594.532819] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:12 2019][2937594.538922] [] schedule+0x29/0x70 [Fri Apr 12 11:11:12 2019][2937594.544073] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:12 2019][2937594.551393] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:12 2019][2937594.557171] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.564494] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937594.571050] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.578298] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.585542] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.592864] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:12 2019][2937594.599849] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.607617] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.614862] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.622629] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.630286] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:12 2019][2937594.636736] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:12 2019][2937594.642756] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937594.649323] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.656118] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:12 2019][2937594.662575] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:12 2019][2937594.668243] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:12 2019][2937594.675841] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:12 2019][2937594.680919] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.687214] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:12 2019][2937594.693863] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.700178] osp-pre-8-2 S ffff8b5da99ca080 0 117655 2 0x00000080 [Fri Apr 12 11:11:12 2019][2937594.707564] Call Trace: [Fri Apr 12 11:11:12 2019][2937594.710197] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:12 2019][2937594.715604] [] schedule+0x29/0x70 [Fri Apr 12 11:11:12 2019][2937594.720781] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:12 2019][2937594.727756] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:12 2019][2937594.733525] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:12 2019][2937594.740253] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:12 2019][2937594.745324] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.751598] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:12 2019][2937594.758218] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:12 2019][2937594.764494] osp-syn-35-2 S ffff8b5da99cd140 0 117656 2 0x00000080 [Fri Apr 12 11:11:12 2019][2937594.771882] Call Trace: [Fri Apr 12 11:11:12 2019][2937594.774515] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:12 2019][2937594.780613] [] schedule+0x29/0x70 [Fri Apr 12 11:11:12 2019][2937594.785770] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:12 2019][2937594.793106] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:12 2019][2937594.798884] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.806205] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:12 2019][2937594.812785] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.820046] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.827295] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.834626] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.842399] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.849645] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:12 2019][2937594.857412] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937594.865073] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:13 2019][2937594.871520] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:13 2019][2937594.877539] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:13 2019][2937594.884089] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:13 2019][2937594.890886] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:13 2019][2937594.897339] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:13 2019][2937594.903008] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:13 2019][2937594.910579] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937594.915663] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937594.921936] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937594.928563] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937594.934856] osp-pre-35-2 S ffff8b6b8e03a080 0 117657 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937594.942218] Call Trace: [Fri Apr 12 11:11:13 2019][2937594.944849] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:13 2019][2937594.950258] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937594.955411] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:13 2019][2937594.962383] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937594.968142] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:13 2019][2937594.974843] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937594.979907] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937594.986182] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937594.992802] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937594.999075] osp-syn-28-2 S ffff8b6b8e03e180 0 117658 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.006431] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.009110] [] ? lustre_swab_niobuf_remote+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:13 2019][2937595.016689] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.021841] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:13 2019][2937595.029152] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937595.034949] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.042267] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:13 2019][2937595.048818] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.056066] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.063309] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.070628] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:13 2019][2937595.077611] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.085373] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.092620] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.100386] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.108049] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:13 2019][2937595.114493] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:13 2019][2937595.120513] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:13 2019][2937595.127062] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.133861] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:13 2019][2937595.140314] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:13 2019][2937595.145986] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:13 2019][2937595.153557] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937595.158617] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.164892] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937595.171513] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.177787] osp-pre-28-2 S ffff8b6b8e03d140 0 117659 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.185151] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.187783] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.192933] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:13 2019][2937595.199908] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937595.205664] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:13 2019][2937595.212369] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937595.217427] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.223695] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937595.230316] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.236582] osp-syn-30-2 S ffff8b6b8e038000 0 117660 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.243938] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.246577] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:13 2019][2937595.253372] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:13 2019][2937595.259471] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.264627] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:13 2019][2937595.271956] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937595.277729] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.285046] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:13 2019][2937595.291596] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.298843] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.306085] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.313421] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.321185] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.328430] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.336196] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.343855] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:13 2019][2937595.350301] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:13 2019][2937595.356322] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:13 2019][2937595.362869] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.369676] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:13 2019][2937595.376121] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:13 2019][2937595.381792] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:13 2019][2937595.389364] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937595.394423] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.400701] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937595.407319] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.413584] osp-pre-30-2 S ffff8b6b8e039040 0 117661 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.420942] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.423574] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.428723] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:13 2019][2937595.435697] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937595.441457] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:13 2019][2937595.448159] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937595.453220] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.459496] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937595.466118] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.472408] osp-syn-34-2 S ffff8b6b8e03b0c0 0 117662 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.479762] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.482403] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:13 2019][2937595.489197] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:13 2019][2937595.495298] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.500452] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:13 2019][2937595.507762] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937595.513555] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.520870] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:13 2019][2937595.527419] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.534662] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.541909] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.549259] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.557032] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.564280] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.572046] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.579705] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:13 2019][2937595.586154] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:13 2019][2937595.592180] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:13 2019][2937595.598729] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:13 2019][2937595.605529] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:13 2019][2937595.611972] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:13 2019][2937595.617645] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:13 2019][2937595.625217] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937595.630274] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.636551] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937595.643173] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.649445] osp-pre-34-2 S ffff8b5da98fb0c0 0 117663 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.656800] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.659431] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.664584] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:13 2019][2937595.671548] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937595.677308] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:13 2019][2937595.684009] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937595.689070] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.695337] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937595.701963] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.708232] ldlm_cn03_003 S ffff8b6c010a8000 0 117680 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.715590] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.718218] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.723415] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:13 2019][2937595.730291] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937595.736091] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:13 2019][2937595.742597] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:13 2019][2937595.750175] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937595.755235] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.761510] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937595.768133] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.774404] ldlm_cn01_003 S ffff8b6c010ac100 0 117681 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.781761] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.784391] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.789583] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:13 2019][2937595.796466] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:13 2019][2937595.802259] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:13 2019][2937595.808739] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:13 2019][2937595.816311] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:13 2019][2937595.821371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.827651] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:13 2019][2937595.834270] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:13 2019][2937595.840542] ldlm_cn01_004 S ffff8b6c010aa080 0 117682 2 0x00000080 [Fri Apr 12 11:11:13 2019][2937595.847904] Call Trace: [Fri Apr 12 11:11:13 2019][2937595.850535] [] schedule+0x29/0x70 [Fri Apr 12 11:11:13 2019][2937595.855725] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:13 2019][2937595.862600] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937595.868391] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937595.874874] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937595.882446] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937595.887507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937595.893774] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937595.900398] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937595.906672] ldlm_cn00_003 S ffff8b6c010ab0c0 0 117683 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937595.914033] Call Trace: [Fri Apr 12 11:11:14 2019][2937595.916664] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937595.921852] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937595.928745] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937595.934537] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937595.941023] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937595.948591] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937595.953670] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937595.959945] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937595.966569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937595.972841] ldlm_cn02_003 S ffff8b4e297a6180 0 117684 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937595.980216] Call Trace: [Fri Apr 12 11:11:14 2019][2937595.982884] [] ? lprocfs_counter_add+0x5c/0x160 [obdclass] [Fri Apr 12 11:11:14 2019][2937595.990208] [] ? schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937595.995563] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.002615] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.008407] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.015063] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.022633] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.027867] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.034143] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.040937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.047212] ldlm_cn02_004 S ffff8b4e297a5140 0 117685 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.054575] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.057206] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.062393] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.069269] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.075063] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.081544] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.089134] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.094196] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.100471] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.107091] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.113365] ldlm_cn00_004 S ffff8b4e297a30c0 0 117686 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.120722] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.123353] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.128551] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.135434] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.141224] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.147710] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.155290] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.160384] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.166661] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.173300] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.179574] ldlm_cn01_005 S ffff8b62607bc100 0 117687 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.186936] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.189566] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.194754] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.201630] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.207422] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.213906] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.221478] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.226540] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.232814] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.239434] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.245708] ldlm_cn01_006 S ffff8b62607b9040 0 117688 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.253063] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.255696] [] ? __schedule+0x725/0x890 [Fri Apr 12 11:11:14 2019][2937596.261370] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.266559] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.273435] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.279237] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.285716] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.293291] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.298352] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.304627] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.311249] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.317523] ldlm_cn03_004 S ffff8b62607ba080 0 117689 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.324885] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.327516] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.332701] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.339597] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.345400] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.351884] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.359455] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.364514] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.370798] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.377420] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.383693] ldlm_cn00_005 S ffff8b62607bd140 0 117690 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.391049] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.393680] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.398868] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.405743] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.411538] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.418019] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.425590] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.430650] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.436917] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.443539] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.449813] lquota_wb_fir-M S ffff8b6c16292080 0 117693 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.457169] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.459799] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:11:14 2019][2937596.465466] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.470612] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:11:14 2019][2937596.476621] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:11:14 2019][2937596.483162] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Fri Apr 12 11:11:14 2019][2937596.489958] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Fri Apr 12 11:11:14 2019][2937596.496571] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.502332] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Fri Apr 12 11:11:14 2019][2937596.508707] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.513767] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.520042] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.526680] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.532955] lquota_wb_fir-M S ffff8b6c16291040 0 117696 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.540317] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.542950] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:11:14 2019][2937596.548618] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.553765] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:11:14 2019][2937596.559780] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:11:14 2019][2937596.566322] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Fri Apr 12 11:11:14 2019][2937596.573114] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Fri Apr 12 11:11:14 2019][2937596.579730] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.585510] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Fri Apr 12 11:11:14 2019][2937596.591864] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.596927] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.603201] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.609822] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.616095] dist_txn-2 S ffff8b6c16295140 0 117697 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.623451] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.626090] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:11:14 2019][2937596.632884] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:14 2019][2937596.638995] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.644198] [] distribute_txn_commit_thread+0x9ac/0xca0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.652039] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.657838] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.664890] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.669953] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.676246] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.682866] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.689142] ldlm_cn02_005 S ffff8b6cfaccb0c0 0 117702 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.696505] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.699134] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.704322] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.711200] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.716994] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.723480] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.731056] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.736115] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.742391] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.749014] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.755285] qmt_reba_fir-QM S ffff8b8cf1ef2080 0 117710 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.762642] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.765272] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:14 2019][2937596.771373] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.776530] [] qmt_reba_thread+0x9e0/0xda0 [lquota] [Fri Apr 12 11:11:14 2019][2937596.783257] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.789015] [] ? qmt_glimpse_lock.isra.19+0x690/0x690 [lquota] [Fri Apr 12 11:11:14 2019][2937596.796670] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:14 2019][2937596.801733] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.808008] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:14 2019][2937596.814629] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:14 2019][2937596.820904] osp_up2-0 S ffff8b8cf1ef5140 0 117713 2 0x00000080 [Fri Apr 12 11:11:14 2019][2937596.828266] Call Trace: [Fri Apr 12 11:11:14 2019][2937596.830942] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Fri Apr 12 11:11:14 2019][2937596.838002] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:11:14 2019][2937596.844973] [] schedule+0x29/0x70 [Fri Apr 12 11:11:14 2019][2937596.850127] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:11:14 2019][2937596.857185] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:14 2019][2937596.862946] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:11:15 2019][2937596.870175] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937596.875236] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937596.881527] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937596.888150] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937596.894425] osp-pre-2-0 S ffff8b881cb8c100 0 117714 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937596.901787] Call Trace: [Fri Apr 12 11:11:15 2019][2937596.904436] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:15 2019][2937596.909843] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937596.914999] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:15 2019][2937596.921970] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937596.927729] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:15 2019][2937596.934441] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937596.939500] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937596.945777] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937596.952398] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937596.958670] osp_up1-0 S ffff8b881cb88000 0 117716 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937596.966026] Call Trace: [Fri Apr 12 11:11:15 2019][2937596.968715] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Fri Apr 12 11:11:15 2019][2937596.975778] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:11:15 2019][2937596.982740] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937596.987894] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:11:15 2019][2937596.994944] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.000703] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:11:15 2019][2937597.007944] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937597.013007] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.019280] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937597.025901] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.032174] osp-pre-1-0 S ffff8b881cb8b0c0 0 117717 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937597.039546] Call Trace: [Fri Apr 12 11:11:15 2019][2937597.042178] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:15 2019][2937597.047588] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937597.052739] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:15 2019][2937597.059703] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.065460] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:15 2019][2937597.072165] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937597.077226] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.083500] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937597.090122] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.096394] osp_up3-0 S ffff8b881cb8d140 0 117718 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937597.103752] Call Trace: [Fri Apr 12 11:11:15 2019][2937597.106427] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Fri Apr 12 11:11:15 2019][2937597.113486] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:11:15 2019][2937597.120447] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937597.125601] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:11:15 2019][2937597.132662] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.138417] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:11:15 2019][2937597.145644] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937597.150702] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.156969] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937597.163608] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.169883] osp-pre-3-0 S ffff8b881cb8a080 0 117719 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937597.177256] Call Trace: [Fri Apr 12 11:11:15 2019][2937597.179886] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:15 2019][2937597.185291] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937597.190437] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:15 2019][2937597.197401] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.203160] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:15 2019][2937597.209864] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937597.214935] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.221207] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937597.227832] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.234105] osp-syn-12-0 S ffff8b881cb89040 0 117720 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937597.241469] Call Trace: [Fri Apr 12 11:11:15 2019][2937597.244098] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:15 2019][2937597.250201] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937597.255354] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:15 2019][2937597.262665] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.268442] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.275756] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:15 2019][2937597.282311] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.289557] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.296803] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.304125] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:15 2019][2937597.311110] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.318875] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.326125] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.333888] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.341548] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:15 2019][2937597.347997] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:15 2019][2937597.354016] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:15 2019][2937597.360565] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.367364] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:15 2019][2937597.373810] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:15 2019][2937597.379480] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:15 2019][2937597.387049] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937597.392111] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.398386] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937597.405009] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.411299] osp-pre-12-0 S ffff8b881cb8e180 0 117721 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937597.418661] Call Trace: [Fri Apr 12 11:11:15 2019][2937597.421292] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:15 2019][2937597.426700] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937597.431861] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:15 2019][2937597.438826] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.444585] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:15 2019][2937597.451287] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937597.456366] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.462643] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937597.469264] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.475538] osp-syn-13-0 S ffff8b6cf33f5140 0 117722 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937597.482899] Call Trace: [Fri Apr 12 11:11:15 2019][2937597.485531] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:15 2019][2937597.491632] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937597.496786] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:15 2019][2937597.504096] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.509872] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.517188] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:15 2019][2937597.523736] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.530983] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.538228] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.545548] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:15 2019][2937597.552538] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.560318] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.567566] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.575332] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.582994] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:15 2019][2937597.589438] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:15 2019][2937597.595459] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:15 2019][2937597.602010] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.608811] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:15 2019][2937597.615267] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:15 2019][2937597.620937] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:15 2019][2937597.628510] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937597.633570] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.639845] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937597.646474] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.652748] osp-pre-13-0 S ffff8b6cf33f6180 0 117723 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937597.660111] Call Trace: [Fri Apr 12 11:11:15 2019][2937597.662745] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:15 2019][2937597.668154] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937597.673305] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:15 2019][2937597.680276] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.686051] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:15 2019][2937597.692765] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:15 2019][2937597.697824] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.704101] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:15 2019][2937597.710721] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:15 2019][2937597.716996] osp-syn-39-0 S ffff8b6cf33f30c0 0 117724 2 0x00000080 [Fri Apr 12 11:11:15 2019][2937597.724361] Call Trace: [Fri Apr 12 11:11:15 2019][2937597.726989] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:15 2019][2937597.733092] [] schedule+0x29/0x70 [Fri Apr 12 11:11:15 2019][2937597.738246] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:15 2019][2937597.745570] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:15 2019][2937597.751350] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.758674] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:15 2019][2937597.765222] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.772470] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.779715] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.787034] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:15 2019][2937597.794028] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.801798] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.809041] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.816809] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.824469] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:15 2019][2937597.830914] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:15 2019][2937597.836936] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:15 2019][2937597.843492] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:15 2019][2937597.850290] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:15 2019][2937597.856735] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:15 2019][2937597.862405] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:16 2019][2937597.869977] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937597.875038] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937597.881328] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937597.887942] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937597.894215] osp-pre-39-0 S ffff8b6cf33f4100 0 117725 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937597.901581] Call Trace: [Fri Apr 12 11:11:16 2019][2937597.904211] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:16 2019][2937597.909618] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937597.914771] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:16 2019][2937597.921737] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937597.927494] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:16 2019][2937597.934198] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937597.939291] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937597.945561] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937597.952180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937597.958455] osp-syn-46-0 S ffff8b6cf33f1040 0 117726 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937597.965819] Call Trace: [Fri Apr 12 11:11:16 2019][2937597.968450] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:16 2019][2937597.974548] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937597.979696] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:16 2019][2937597.987006] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937597.992781] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.000096] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:16 2019][2937598.006645] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.013892] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.021136] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.028468] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.036233] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.043479] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.051245] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.058905] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:16 2019][2937598.065352] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:16 2019][2937598.071375] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:16 2019][2937598.077922] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.084730] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:16 2019][2937598.091180] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:16 2019][2937598.096851] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:16 2019][2937598.104423] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937598.109482] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.115752] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937598.122373] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.128645] osp-pre-46-0 S ffff8b6cf33f2080 0 117727 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937598.136005] Call Trace: [Fri Apr 12 11:11:16 2019][2937598.138643] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:16 2019][2937598.144049] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937598.149201] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:16 2019][2937598.156176] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937598.161933] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:16 2019][2937598.168636] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937598.173697] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.179972] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937598.186594] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.192885] osp-syn-10-0 S ffff8b6b4ef10000 0 117728 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937598.200249] Call Trace: [Fri Apr 12 11:11:16 2019][2937598.202881] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:16 2019][2937598.208981] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937598.214144] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:16 2019][2937598.221452] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937598.227247] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.234561] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:16 2019][2937598.241111] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.248367] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.255628] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.262963] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.270724] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.277979] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.285751] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.293415] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:16 2019][2937598.299861] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:16 2019][2937598.305879] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:16 2019][2937598.312432] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.319238] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:16 2019][2937598.325696] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:16 2019][2937598.331371] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:16 2019][2937598.338949] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937598.344013] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.350284] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937598.356906] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.363181] osp-pre-10-0 S ffff8b6b4ef12080 0 117729 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937598.370552] Call Trace: [Fri Apr 12 11:11:16 2019][2937598.373185] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:16 2019][2937598.378590] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937598.383759] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:16 2019][2937598.390726] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937598.396483] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:16 2019][2937598.403188] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937598.408248] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.414524] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937598.421145] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.427418] osp-syn-11-0 S ffff8b6b4ef15140 0 117730 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937598.434774] Call Trace: [Fri Apr 12 11:11:16 2019][2937598.437405] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:16 2019][2937598.443506] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937598.448660] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:16 2019][2937598.455971] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937598.461745] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.469061] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:16 2019][2937598.475608] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.482856] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.490100] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.497433] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.505196] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.512446] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.520207] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.527869] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:16 2019][2937598.534317] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:16 2019][2937598.540347] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:16 2019][2937598.546894] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.553694] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:16 2019][2937598.560137] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:16 2019][2937598.565808] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:16 2019][2937598.573379] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937598.578455] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.584724] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937598.591345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.597618] osp-pre-11-0 S ffff8b6b4ef11040 0 117731 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937598.604983] Call Trace: [Fri Apr 12 11:11:16 2019][2937598.607614] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:16 2019][2937598.613022] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937598.618175] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:16 2019][2937598.625138] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937598.630897] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:16 2019][2937598.637600] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937598.642661] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.648935] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937598.655558] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.661832] osp-syn-14-0 S ffff8b6b4ef14100 0 117732 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937598.669194] Call Trace: [Fri Apr 12 11:11:16 2019][2937598.671826] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:16 2019][2937598.677927] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937598.683080] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:16 2019][2937598.690389] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937598.696165] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.703482] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:16 2019][2937598.710033] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.717287] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.724531] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.731864] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.739627] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.746874] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.754641] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.762309] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:16 2019][2937598.768756] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:16 2019][2937598.774776] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:16 2019][2937598.781324] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:16 2019][2937598.788123] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:16 2019][2937598.794567] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:16 2019][2937598.800239] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:16 2019][2937598.807811] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:16 2019][2937598.812871] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.819145] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:16 2019][2937598.825766] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:16 2019][2937598.832040] osp-pre-14-0 S ffff8b6b4ef130c0 0 117733 2 0x00000080 [Fri Apr 12 11:11:16 2019][2937598.839412] Call Trace: [Fri Apr 12 11:11:16 2019][2937598.842045] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:16 2019][2937598.847453] [] schedule+0x29/0x70 [Fri Apr 12 11:11:16 2019][2937598.852604] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:16 2019][2937598.859569] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:16 2019][2937598.865325] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:17 2019][2937598.872033] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937598.877091] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937598.883369] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937598.889990] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937598.896262] osp-syn-26-0 S ffff8b5da98b8000 0 117734 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937598.903625] Call Trace: [Fri Apr 12 11:11:17 2019][2937598.906257] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:17 2019][2937598.912356] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937598.917510] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:17 2019][2937598.924820] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937598.930598] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937598.937911] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:17 2019][2937598.944470] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:17 2019][2937598.951734] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:17 2019][2937598.958979] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937598.966312] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:17 2019][2937598.974077] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:17 2019][2937598.981324] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:17 2019][2937598.989106] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937598.996767] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:17 2019][2937599.003213] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:17 2019][2937599.009232] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:17 2019][2937599.015784] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.022580] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:17 2019][2937599.029025] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:17 2019][2937599.034694] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:17 2019][2937599.042284] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937599.047343] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.053618] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937599.060240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.066513] osp-pre-26-0 S ffff8b5da98bb0c0 0 117735 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937599.073870] Call Trace: [Fri Apr 12 11:11:17 2019][2937599.076501] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:17 2019][2937599.081907] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937599.087062] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:17 2019][2937599.094036] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937599.099792] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:17 2019][2937599.106494] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937599.111555] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.117830] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937599.124453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.130726] osp-syn-31-0 S ffff8b5da99630c0 0 117736 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937599.138091] Call Trace: [Fri Apr 12 11:11:17 2019][2937599.140722] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:17 2019][2937599.146821] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937599.151975] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:17 2019][2937599.159285] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937599.165061] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.172376] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:17 2019][2937599.178923] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.186171] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.193425] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.200759] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.208520] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.215769] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.223535] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.231196] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:17 2019][2937599.237640] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:17 2019][2937599.243664] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:17 2019][2937599.250221] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.257016] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:17 2019][2937599.263462] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:17 2019][2937599.269131] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:17 2019][2937599.276703] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937599.281763] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.288033] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937599.294651] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.300918] osp-pre-31-0 S ffff8b6cf6826180 0 117737 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937599.308282] Call Trace: [Fri Apr 12 11:11:17 2019][2937599.310914] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:17 2019][2937599.316319] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937599.321476] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:17 2019][2937599.328443] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937599.334204] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:17 2019][2937599.340907] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937599.345968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.352242] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937599.358864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.365138] osp-syn-42-0 S ffff8b6cf6822080 0 117738 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937599.372496] Call Trace: [Fri Apr 12 11:11:17 2019][2937599.375124] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:17 2019][2937599.381226] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937599.386378] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:17 2019][2937599.393690] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937599.399463] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.406780] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:17 2019][2937599.413330] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.420577] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.427822] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.435154] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.442917] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.450164] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.457930] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.465591] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:17 2019][2937599.472040] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:17 2019][2937599.478057] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:17 2019][2937599.484606] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.491403] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:17 2019][2937599.497849] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:17 2019][2937599.503521] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:17 2019][2937599.511089] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937599.516149] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.522418] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937599.529039] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.535312] osp-pre-42-0 S ffff8b6cf6825140 0 117739 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937599.542677] Call Trace: [Fri Apr 12 11:11:17 2019][2937599.545307] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:17 2019][2937599.550717] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937599.555868] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:17 2019][2937599.562833] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937599.568593] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:17 2019][2937599.575305] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937599.580365] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.586659] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937599.593289] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.599570] osp-syn-9-0 S ffff8b6cf6820000 0 117740 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937599.606942] Call Trace: [Fri Apr 12 11:11:17 2019][2937599.609574] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:17 2019][2937599.615674] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937599.620828] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:17 2019][2937599.628137] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937599.633916] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.641245] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:17 2019][2937599.647794] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.655044] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.662293] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.669643] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.677409] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.684656] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.692418] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.700083] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:17 2019][2937599.706527] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:17 2019][2937599.712548] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:17 2019][2937599.719099] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:17 2019][2937599.725895] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:17 2019][2937599.732350] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:17 2019][2937599.738019] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:17 2019][2937599.745606] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937599.750678] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.756969] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937599.763590] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.769864] osp-pre-9-0 S ffff8b6cf6821040 0 117741 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937599.777229] Call Trace: [Fri Apr 12 11:11:17 2019][2937599.779859] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:17 2019][2937599.785267] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937599.790421] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:17 2019][2937599.797383] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937599.803141] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:17 2019][2937599.809844] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:17 2019][2937599.814907] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.821183] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:17 2019][2937599.827803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:17 2019][2937599.834076] osp-syn-2-0 S ffff8b6cf6824100 0 117742 2 0x00000080 [Fri Apr 12 11:11:17 2019][2937599.841430] Call Trace: [Fri Apr 12 11:11:17 2019][2937599.844063] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:17 2019][2937599.850164] [] schedule+0x29/0x70 [Fri Apr 12 11:11:17 2019][2937599.855319] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:17 2019][2937599.862646] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:17 2019][2937599.868420] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.875736] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937599.882284] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.889532] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.896778] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.904110] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.911874] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.919121] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.926883] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.934546] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:18 2019][2937599.940993] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:18 2019][2937599.947011] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937599.953564] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:18 2019][2937599.960361] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:18 2019][2937599.966806] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:18 2019][2937599.972475] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:18 2019][2937599.980051] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:18 2019][2937599.985117] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937599.991390] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:18 2019][2937599.998011] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.004286] osp-pre-2-0 S ffff8b6cfcdd0000 0 117743 2 0x00000080 [Fri Apr 12 11:11:18 2019][2937600.011649] Call Trace: [Fri Apr 12 11:11:18 2019][2937600.014283] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:18 2019][2937600.019687] [] schedule+0x29/0x70 [Fri Apr 12 11:11:18 2019][2937600.024841] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:18 2019][2937600.031804] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:18 2019][2937600.037564] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:18 2019][2937600.044268] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:18 2019][2937600.049327] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.055595] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:18 2019][2937600.062217] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.068489] osp-syn-43-0 S ffff8b6cfcdd4100 0 117744 2 0x00000080 [Fri Apr 12 11:11:18 2019][2937600.075854] Call Trace: [Fri Apr 12 11:11:18 2019][2937600.078486] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:18 2019][2937600.084586] [] schedule+0x29/0x70 [Fri Apr 12 11:11:18 2019][2937600.089741] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:18 2019][2937600.097058] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:18 2019][2937600.102835] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.110167] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937600.116713] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.123961] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.131208] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.138539] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.146304] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.153551] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.161316] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.168985] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:18 2019][2937600.175431] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:18 2019][2937600.181450] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937600.188000] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.194798] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:18 2019][2937600.201252] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:18 2019][2937600.206923] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:18 2019][2937600.214494] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:18 2019][2937600.219552] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.225838] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:18 2019][2937600.232459] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.238733] osp-pre-43-0 S ffff8b6ee1366180 0 117745 2 0x00000080 [Fri Apr 12 11:11:18 2019][2937600.246097] Call Trace: [Fri Apr 12 11:11:18 2019][2937600.248730] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:18 2019][2937600.254134] [] schedule+0x29/0x70 [Fri Apr 12 11:11:18 2019][2937600.259289] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:18 2019][2937600.266251] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:18 2019][2937600.272019] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:18 2019][2937600.278723] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:18 2019][2937600.283800] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.290067] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:18 2019][2937600.296698] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.302981] osp-syn-19-0 S ffff8b6ee1365140 0 117746 2 0x00000080 [Fri Apr 12 11:11:18 2019][2937600.310345] Call Trace: [Fri Apr 12 11:11:18 2019][2937600.312975] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:18 2019][2937600.319076] [] schedule+0x29/0x70 [Fri Apr 12 11:11:18 2019][2937600.324230] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:18 2019][2937600.331539] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:18 2019][2937600.337316] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.344632] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937600.351181] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.358425] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.365669] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.373005] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.380768] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.388015] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.395781] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.403439] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:18 2019][2937600.409890] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:18 2019][2937600.415924] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937600.422475] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.429273] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:18 2019][2937600.435735] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:18 2019][2937600.441406] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:18 2019][2937600.448974] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:18 2019][2937600.454036] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.460312] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:18 2019][2937600.466931] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.473208] osp-pre-19-0 S ffff8b6ee1364100 0 117747 2 0x00000080 [Fri Apr 12 11:11:18 2019][2937600.480570] Call Trace: [Fri Apr 12 11:11:18 2019][2937600.483202] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:18 2019][2937600.488608] [] schedule+0x29/0x70 [Fri Apr 12 11:11:18 2019][2937600.493764] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:18 2019][2937600.500734] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:18 2019][2937600.506491] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:18 2019][2937600.513197] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:18 2019][2937600.518256] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.524525] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:18 2019][2937600.531144] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.537418] osp-syn-18-0 S ffff8b6ee13630c0 0 117748 2 0x00000080 [Fri Apr 12 11:11:18 2019][2937600.544775] Call Trace: [Fri Apr 12 11:11:18 2019][2937600.547406] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:18 2019][2937600.553504] [] schedule+0x29/0x70 [Fri Apr 12 11:11:18 2019][2937600.558658] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:18 2019][2937600.565968] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:18 2019][2937600.571745] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.579079] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937600.585635] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.592880] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.600123] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.607460] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.615221] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.622470] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.630238] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.637895] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:18 2019][2937600.644342] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:18 2019][2937600.650361] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937600.656913] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.663712] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:18 2019][2937600.670179] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:18 2019][2937600.675852] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:18 2019][2937600.683422] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:18 2019][2937600.688482] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.694758] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:18 2019][2937600.701380] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.707653] osp-pre-18-0 S ffff8b6ee1361040 0 117749 2 0x00000080 [Fri Apr 12 11:11:18 2019][2937600.715009] Call Trace: [Fri Apr 12 11:11:18 2019][2937600.717641] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:18 2019][2937600.723046] [] schedule+0x29/0x70 [Fri Apr 12 11:11:18 2019][2937600.728215] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:18 2019][2937600.735181] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:18 2019][2937600.740941] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:18 2019][2937600.747643] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:18 2019][2937600.752704] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.758982] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:18 2019][2937600.765600] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:18 2019][2937600.771875] osp-syn-25-0 S ffff8b742a2430c0 0 117750 2 0x00000080 [Fri Apr 12 11:11:18 2019][2937600.779239] Call Trace: [Fri Apr 12 11:11:18 2019][2937600.781871] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:18 2019][2937600.787970] [] schedule+0x29/0x70 [Fri Apr 12 11:11:18 2019][2937600.793123] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:18 2019][2937600.800434] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:18 2019][2937600.806210] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.813527] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:18 2019][2937600.820084] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.827328] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.834574] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.841909] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.849672] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.856918] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:18 2019][2937600.864699] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937600.872360] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:19 2019][2937600.878826] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:19 2019][2937600.884844] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937600.891395] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:19 2019][2937600.898192] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:19 2019][2937600.904636] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:19 2019][2937600.910307] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:19 2019][2937600.917879] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937600.922941] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937600.929213] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:19 2019][2937600.935847] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937600.942118] osp-pre-25-0 S ffff8b742a245140 0 117751 2 0x00000080 [Fri Apr 12 11:11:19 2019][2937600.949482] Call Trace: [Fri Apr 12 11:11:19 2019][2937600.952112] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:19 2019][2937600.957521] [] schedule+0x29/0x70 [Fri Apr 12 11:11:19 2019][2937600.962674] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:19 2019][2937600.969638] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:19 2019][2937600.975395] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:19 2019][2937600.982100] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937600.987162] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937600.993446] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:19 2019][2937601.000071] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.006339] osp-syn-17-0 S ffff8b742a240000 0 117752 2 0x00000080 [Fri Apr 12 11:11:19 2019][2937601.013694] Call Trace: [Fri Apr 12 11:11:19 2019][2937601.016327] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:19 2019][2937601.022428] [] schedule+0x29/0x70 [Fri Apr 12 11:11:19 2019][2937601.027580] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:19 2019][2937601.034890] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:19 2019][2937601.040666] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.047983] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937601.054539] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.061794] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.069039] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.076372] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.084139] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.091384] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.099149] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.106810] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:19 2019][2937601.113270] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:19 2019][2937601.119282] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937601.125833] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.132632] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:19 2019][2937601.139077] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:19 2019][2937601.144746] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:19 2019][2937601.152316] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937601.157379] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.163654] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:19 2019][2937601.170273] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.176555] osp-pre-17-0 S ffff8b742a242080 0 117753 2 0x00000080 [Fri Apr 12 11:11:19 2019][2937601.183912] Call Trace: [Fri Apr 12 11:11:19 2019][2937601.186543] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:19 2019][2937601.191949] [] schedule+0x29/0x70 [Fri Apr 12 11:11:19 2019][2937601.197104] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:19 2019][2937601.204071] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:19 2019][2937601.209825] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:19 2019][2937601.216530] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937601.221591] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.227867] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:19 2019][2937601.234485] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.240753] osp-syn-0-0 S ffff8b742a241040 0 117754 2 0x00000080 [Fri Apr 12 11:11:19 2019][2937601.248115] Call Trace: [Fri Apr 12 11:11:19 2019][2937601.250748] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:19 2019][2937601.256847] [] schedule+0x29/0x70 [Fri Apr 12 11:11:19 2019][2937601.262002] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:19 2019][2937601.269311] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:19 2019][2937601.275093] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.282409] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937601.288961] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.296208] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.303452] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.310789] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.318558] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.325804] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.333571] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.341230] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:19 2019][2937601.347677] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:19 2019][2937601.353695] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937601.360247] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.367044] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:19 2019][2937601.373489] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:19 2019][2937601.379157] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:19 2019][2937601.386729] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937601.391790] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.398069] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:19 2019][2937601.404686] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.410951] osp-pre-0-0 S ffff8b742a244100 0 117755 2 0x00000080 [Fri Apr 12 11:11:19 2019][2937601.418306] Call Trace: [Fri Apr 12 11:11:19 2019][2937601.420939] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:19 2019][2937601.426347] [] schedule+0x29/0x70 [Fri Apr 12 11:11:19 2019][2937601.431500] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:19 2019][2937601.438462] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:19 2019][2937601.444222] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:19 2019][2937601.450924] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937601.455988] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.462260] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:19 2019][2937601.468884] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.475156] osp-syn-5-0 S ffff8b742a246180 0 117756 2 0x00000080 [Fri Apr 12 11:11:19 2019][2937601.482512] Call Trace: [Fri Apr 12 11:11:19 2019][2937601.485143] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:19 2019][2937601.491241] [] schedule+0x29/0x70 [Fri Apr 12 11:11:19 2019][2937601.496389] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:19 2019][2937601.503697] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:19 2019][2937601.509473] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.516789] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937601.523342] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.530587] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.537831] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.545162] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.552928] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.560173] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.567940] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.575600] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:19 2019][2937601.582047] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:19 2019][2937601.588067] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937601.594615] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.601414] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:19 2019][2937601.607857] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:19 2019][2937601.613529] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:19 2019][2937601.621112] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937601.626168] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.632443] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:19 2019][2937601.639065] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.645338] osp-pre-5-0 S ffff8b7cf328b0c0 0 117757 2 0x00000080 [Fri Apr 12 11:11:19 2019][2937601.652693] Call Trace: [Fri Apr 12 11:11:19 2019][2937601.655326] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:19 2019][2937601.660733] [] schedule+0x29/0x70 [Fri Apr 12 11:11:19 2019][2937601.665886] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:19 2019][2937601.672850] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:19 2019][2937601.678607] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:19 2019][2937601.685312] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937601.690373] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.696647] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:19 2019][2937601.703268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.709544] osp-syn-44-0 S ffff8b7cf3288000 0 117758 2 0x00000080 [Fri Apr 12 11:11:19 2019][2937601.716907] Call Trace: [Fri Apr 12 11:11:19 2019][2937601.719538] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:19 2019][2937601.725637] [] schedule+0x29/0x70 [Fri Apr 12 11:11:19 2019][2937601.730792] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:19 2019][2937601.738102] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:19 2019][2937601.743877] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.751195] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937601.757750] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.764998] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.772240] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.779576] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.787338] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.794587] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.802351] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.810010] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:19 2019][2937601.816458] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:19 2019][2937601.822479] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:19 2019][2937601.829028] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:19 2019][2937601.835827] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:19 2019][2937601.842271] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:19 2019][2937601.847942] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:19 2019][2937601.855511] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:19 2019][2937601.860572] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:19 2019][2937601.866849] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937601.873470] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937601.879741] osp-pre-44-0 S ffff8b7cf328a080 0 117759 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937601.887099] Call Trace: [Fri Apr 12 11:11:20 2019][2937601.889730] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:20 2019][2937601.895135] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937601.900289] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:20 2019][2937601.907253] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937601.913012] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:20 2019][2937601.919716] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937601.924777] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937601.931052] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937601.937673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937601.943946] osp-syn-32-0 S ffff8b7cf3289040 0 117760 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937601.951302] Call Trace: [Fri Apr 12 11:11:20 2019][2937601.953933] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:20 2019][2937601.960037] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937601.965188] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:20 2019][2937601.972498] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937601.978272] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937601.985591] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:20 2019][2937601.992146] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:20 2019][2937601.999395] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.006638] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.013970] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.021735] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.028982] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.036748] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.044406] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:20 2019][2937602.050855] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:20 2019][2937602.056874] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:20 2019][2937602.063429] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.070230] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:20 2019][2937602.076675] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:20 2019][2937602.082346] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:20 2019][2937602.089917] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937602.094977] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.101253] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937602.107874] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.114147] osp-pre-32-0 S ffff8b7cf328e180 0 117761 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937602.121510] Call Trace: [Fri Apr 12 11:11:20 2019][2937602.124143] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:20 2019][2937602.129549] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937602.134703] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:20 2019][2937602.141667] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937602.147426] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:20 2019][2937602.154128] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937602.159189] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.165465] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937602.172092] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.178361] osp-syn-21-0 S ffff8b77c0abb0c0 0 117762 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937602.185724] Call Trace: [Fri Apr 12 11:11:20 2019][2937602.188370] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:20 2019][2937602.194471] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937602.199618] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:20 2019][2937602.206928] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937602.212704] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.220018] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:20 2019][2937602.226567] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.233816] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.241060] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.248392] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.256157] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.263406] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.271171] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.278829] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:20 2019][2937602.285277] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:20 2019][2937602.291297] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:20 2019][2937602.297845] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.304643] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:20 2019][2937602.311090] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:20 2019][2937602.316757] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:20 2019][2937602.324329] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937602.329390] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.335665] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937602.342285] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.348559] osp-pre-21-0 S ffff8b77c0abd140 0 117763 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937602.355915] Call Trace: [Fri Apr 12 11:11:20 2019][2937602.358547] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:20 2019][2937602.363952] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937602.369109] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:20 2019][2937602.376079] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937602.381838] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:20 2019][2937602.388541] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937602.393601] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.399877] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937602.406499] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.412772] osp-syn-16-0 S ffff8b77c0abe180 0 117764 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937602.420128] Call Trace: [Fri Apr 12 11:11:20 2019][2937602.422758] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:20 2019][2937602.428859] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937602.434028] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:20 2019][2937602.441342] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937602.447118] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.454432] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:20 2019][2937602.460981] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.468227] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.475473] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.482805] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.490569] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.497817] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.505580] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.513240] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:20 2019][2937602.519689] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:20 2019][2937602.525707] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:20 2019][2937602.532256] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.539052] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:20 2019][2937602.545499] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:20 2019][2937602.551171] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:20 2019][2937602.558742] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937602.563801] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.570097] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937602.576716] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.582989] osp-pre-16-0 S ffff8b77c0abc100 0 117765 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937602.590345] Call Trace: [Fri Apr 12 11:11:20 2019][2937602.592976] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:20 2019][2937602.598386] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937602.603537] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:20 2019][2937602.610512] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937602.616269] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:20 2019][2937602.622973] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937602.628034] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.634312] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937602.640931] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.647203] osp-syn-41-0 S ffff8b77c0ab8000 0 117766 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937602.654567] Call Trace: [Fri Apr 12 11:11:20 2019][2937602.657200] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:20 2019][2937602.663300] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937602.668475] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:20 2019][2937602.675923] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937602.681738] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.689089] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:20 2019][2937602.695646] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.702910] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.710206] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.717537] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.725304] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.732620] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.740409] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.748082] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:20 2019][2937602.754539] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:20 2019][2937602.760556] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:20 2019][2937602.767148] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:20 2019][2937602.773975] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:20 2019][2937602.780436] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:20 2019][2937602.786108] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:20 2019][2937602.793712] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937602.798791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.805076] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:20 2019][2937602.811723] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.817997] osp-pre-41-0 S ffff8b77c0aba080 0 117767 2 0x00000080 [Fri Apr 12 11:11:20 2019][2937602.825373] Call Trace: [Fri Apr 12 11:11:20 2019][2937602.828017] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:20 2019][2937602.833436] [] schedule+0x29/0x70 [Fri Apr 12 11:11:20 2019][2937602.838596] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:20 2019][2937602.845583] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:20 2019][2937602.851369] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:20 2019][2937602.858098] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:20 2019][2937602.863164] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:20 2019][2937602.869441] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937602.876063] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937602.882334] osp-syn-4-0 S ffff8b77c0ab9040 0 117768 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937602.889691] Call Trace: [Fri Apr 12 11:11:21 2019][2937602.892319] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:21 2019][2937602.898419] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937602.903567] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:21 2019][2937602.910877] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:21 2019][2937602.916654] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937602.923968] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:21 2019][2937602.930517] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:21 2019][2937602.937761] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:21 2019][2937602.945024] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937602.952345] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:11:21 2019][2937602.959325] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:21 2019][2937602.967092] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:21 2019][2937602.974336] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:21 2019][2937602.982105] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937602.989760] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:21 2019][2937602.996200] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:21 2019][2937603.002221] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:21 2019][2937603.008771] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.015568] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:21 2019][2937603.022013] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:21 2019][2937603.027683] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:21 2019][2937603.035253] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:21 2019][2937603.040313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.046581] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937603.053202] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.059477] osp-pre-4-0 S ffff8b7cfe4b8000 0 117769 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937603.066840] Call Trace: [Fri Apr 12 11:11:21 2019][2937603.069471] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:21 2019][2937603.074879] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937603.080034] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:21 2019][2937603.086994] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:21 2019][2937603.092753] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:21 2019][2937603.099458] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:21 2019][2937603.104518] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.110793] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937603.117406] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.123679] osp-syn-23-0 S ffff8b7cfe4bb0c0 0 117770 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937603.131037] Call Trace: [Fri Apr 12 11:11:21 2019][2937603.133666] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:21 2019][2937603.139767] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937603.144920] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:21 2019][2937603.152232] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:21 2019][2937603.158003] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.165321] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:21 2019][2937603.171874] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.179120] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.186363] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.193697] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.201460] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.208709] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.216472] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.224133] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:21 2019][2937603.230577] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:21 2019][2937603.236591] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:21 2019][2937603.243139] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.249936] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:21 2019][2937603.256382] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:21 2019][2937603.262051] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:21 2019][2937603.269624] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:21 2019][2937603.274685] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.280961] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937603.287579] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.293846] osp-pre-23-0 S ffff8b7cfe4bc100 0 117771 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937603.301210] Call Trace: [Fri Apr 12 11:11:21 2019][2937603.303839] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:21 2019][2937603.309248] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937603.314400] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:21 2019][2937603.321364] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:21 2019][2937603.327138] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:21 2019][2937603.333845] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:21 2019][2937603.338905] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.345182] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937603.351802] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.358077] osp-syn-33-0 S ffff8b7cfe4b9040 0 117772 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937603.365441] Call Trace: [Fri Apr 12 11:11:21 2019][2937603.368070] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:21 2019][2937603.374171] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937603.379327] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:21 2019][2937603.386659] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:21 2019][2937603.392426] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.399744] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:21 2019][2937603.406294] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.413541] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.420785] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.428120] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.435888] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.443139] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.450903] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.458576] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:21 2019][2937603.465018] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:21 2019][2937603.471038] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:21 2019][2937603.477587] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.484384] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:21 2019][2937603.490829] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:21 2019][2937603.496500] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:21 2019][2937603.504069] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:21 2019][2937603.509132] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.515408] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937603.522028] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.528301] osp-pre-33-0 S ffff8b7cfe4bd140 0 117773 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937603.535657] Call Trace: [Fri Apr 12 11:11:21 2019][2937603.538289] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:21 2019][2937603.543696] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937603.548849] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:21 2019][2937603.555811] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:21 2019][2937603.561572] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:21 2019][2937603.568276] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:21 2019][2937603.573335] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.579612] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937603.586232] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.592507] osp-syn-20-0 S ffff8b7cfe4be180 0 117774 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937603.599871] Call Trace: [Fri Apr 12 11:11:21 2019][2937603.602500] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:21 2019][2937603.608602] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937603.613756] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:21 2019][2937603.621066] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:21 2019][2937603.626840] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.634157] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:21 2019][2937603.640705] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.647953] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.655212] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.662549] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.670313] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.677557] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.685324] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.692983] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:21 2019][2937603.699430] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:21 2019][2937603.705452] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:21 2019][2937603.712000] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:21 2019][2937603.718797] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:21 2019][2937603.725241] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:21 2019][2937603.730913] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:21 2019][2937603.738483] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:21 2019][2937603.743543] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.749819] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937603.756441] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.762714] osp-pre-20-0 S ffff8b782d21b0c0 0 117775 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937603.770071] Call Trace: [Fri Apr 12 11:11:21 2019][2937603.772700] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:21 2019][2937603.778107] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937603.783261] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:21 2019][2937603.790224] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:21 2019][2937603.795983] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:21 2019][2937603.802688] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:21 2019][2937603.807749] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.814025] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:21 2019][2937603.820645] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:21 2019][2937603.826918] osp-syn-7-0 S ffff8b782d218000 0 117776 2 0x00000080 [Fri Apr 12 11:11:21 2019][2937603.834292] Call Trace: [Fri Apr 12 11:11:21 2019][2937603.836924] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:21 2019][2937603.843023] [] schedule+0x29/0x70 [Fri Apr 12 11:11:21 2019][2937603.848177] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:22 2019][2937603.855485] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937603.861260] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.868579] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937603.875130] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.882372] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.889618] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.896951] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.904717] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.911962] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.919726] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.927387] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:22 2019][2937603.933835] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:22 2019][2937603.939856] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937603.946402] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:22 2019][2937603.953202] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:22 2019][2937603.959646] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:22 2019][2937603.965317] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:22 2019][2937603.972887] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:22 2019][2937603.977947] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937603.984217] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:22 2019][2937603.990838] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937603.997111] osp-pre-7-0 S ffff8b782d21c100 0 117777 2 0x00000080 [Fri Apr 12 11:11:22 2019][2937604.004475] Call Trace: [Fri Apr 12 11:11:22 2019][2937604.007105] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:22 2019][2937604.012513] [] schedule+0x29/0x70 [Fri Apr 12 11:11:22 2019][2937604.017669] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:22 2019][2937604.024638] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937604.030395] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:22 2019][2937604.037100] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:22 2019][2937604.042161] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.048437] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:22 2019][2937604.055058] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.061330] osp-syn-40-0 S ffff8b782d219040 0 117778 2 0x00000080 [Fri Apr 12 11:11:22 2019][2937604.068688] Call Trace: [Fri Apr 12 11:11:22 2019][2937604.071317] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:22 2019][2937604.077420] [] schedule+0x29/0x70 [Fri Apr 12 11:11:22 2019][2937604.082571] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:22 2019][2937604.089882] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937604.095656] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.102974] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937604.109522] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.116769] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.124013] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.131347] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.139111] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.146357] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.154126] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.161791] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:22 2019][2937604.168240] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:22 2019][2937604.174257] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937604.180809] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.187606] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:22 2019][2937604.194050] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:22 2019][2937604.199721] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:22 2019][2937604.207291] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:22 2019][2937604.212351] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.218627] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:22 2019][2937604.225249] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.231522] osp-pre-40-0 S ffff8b782d21a080 0 117779 2 0x00000080 [Fri Apr 12 11:11:22 2019][2937604.238877] Call Trace: [Fri Apr 12 11:11:22 2019][2937604.241509] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:22 2019][2937604.246916] [] schedule+0x29/0x70 [Fri Apr 12 11:11:22 2019][2937604.252070] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:22 2019][2937604.259033] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937604.264791] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:22 2019][2937604.271493] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:22 2019][2937604.276562] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.282831] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:22 2019][2937604.289452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.295726] osp-syn-45-0 S ffff8b782d21d140 0 117780 2 0x00000080 [Fri Apr 12 11:11:22 2019][2937604.303083] Call Trace: [Fri Apr 12 11:11:22 2019][2937604.305712] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:22 2019][2937604.311813] [] schedule+0x29/0x70 [Fri Apr 12 11:11:22 2019][2937604.316967] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:22 2019][2937604.324287] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937604.330062] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.337376] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937604.343928] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.351172] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.358420] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.365750] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.373516] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.380764] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.388530] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.396187] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:22 2019][2937604.402635] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:22 2019][2937604.408670] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937604.415222] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.422020] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:22 2019][2937604.428462] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:22 2019][2937604.434138] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:22 2019][2937604.441713] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:22 2019][2937604.446773] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.453051] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:22 2019][2937604.459669] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.465945] osp-pre-45-0 S ffff8b7cf32b30c0 0 117781 2 0x00000080 [Fri Apr 12 11:11:22 2019][2937604.473309] Call Trace: [Fri Apr 12 11:11:22 2019][2937604.475938] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:22 2019][2937604.481346] [] schedule+0x29/0x70 [Fri Apr 12 11:11:22 2019][2937604.486500] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:22 2019][2937604.493464] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937604.499221] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:22 2019][2937604.505926] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:22 2019][2937604.510985] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.517252] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:22 2019][2937604.523874] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.530149] osp-syn-6-0 S ffff8b7cf32b4100 0 117782 2 0x00000080 [Fri Apr 12 11:11:22 2019][2937604.537521] Call Trace: [Fri Apr 12 11:11:22 2019][2937604.540151] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:22 2019][2937604.546253] [] schedule+0x29/0x70 [Fri Apr 12 11:11:22 2019][2937604.551407] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:22 2019][2937604.558717] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937604.564492] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.571806] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937604.578358] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.585603] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.592849] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.600181] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.607945] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.615191] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.622957] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.630617] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:22 2019][2937604.637067] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:22 2019][2937604.643085] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937604.649632] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.656430] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:22 2019][2937604.662876] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:22 2019][2937604.668547] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:22 2019][2937604.676117] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:22 2019][2937604.681179] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.687454] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:22 2019][2937604.694073] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.700349] osp-pre-6-0 S ffff8b7cf32b6180 0 117783 2 0x00000080 [Fri Apr 12 11:11:22 2019][2937604.707711] Call Trace: [Fri Apr 12 11:11:22 2019][2937604.710343] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:22 2019][2937604.715749] [] schedule+0x29/0x70 [Fri Apr 12 11:11:22 2019][2937604.720919] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:22 2019][2937604.727889] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937604.733648] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:22 2019][2937604.740376] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:22 2019][2937604.745462] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.751740] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:22 2019][2937604.758363] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:22 2019][2937604.764650] osp-syn-24-0 S ffff8b7cf32b2080 0 117784 2 0x00000080 [Fri Apr 12 11:11:22 2019][2937604.772019] Call Trace: [Fri Apr 12 11:11:22 2019][2937604.774655] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:22 2019][2937604.780774] [] schedule+0x29/0x70 [Fri Apr 12 11:11:22 2019][2937604.785938] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:22 2019][2937604.793254] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:22 2019][2937604.799045] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.806370] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:22 2019][2937604.812926] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.820173] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.827419] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.834749] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.842514] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:22 2019][2937604.849763] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:23 2019][2937604.857526] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937604.865186] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:23 2019][2937604.871632] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:23 2019][2937604.877652] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:23 2019][2937604.884201] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:23 2019][2937604.891000] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:23 2019][2937604.897444] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:23 2019][2937604.903116] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:23 2019][2937604.910687] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:23 2019][2937604.915747] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937604.922021] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:23 2019][2937604.928643] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937604.934918] osp-pre-24-0 S ffff8b7cfc304100 0 117785 2 0x00000080 [Fri Apr 12 11:11:23 2019][2937604.942281] Call Trace: [Fri Apr 12 11:11:23 2019][2937604.944911] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:23 2019][2937604.950317] [] schedule+0x29/0x70 [Fri Apr 12 11:11:23 2019][2937604.955465] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:23 2019][2937604.962427] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:23 2019][2937604.968187] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:23 2019][2937604.974906] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:23 2019][2937604.979970] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937604.986244] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:23 2019][2937604.992865] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937604.999139] osp-syn-47-0 S ffff8b7cfc3030c0 0 117786 2 0x00000080 [Fri Apr 12 11:11:23 2019][2937605.006503] Call Trace: [Fri Apr 12 11:11:23 2019][2937605.009134] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:23 2019][2937605.015234] [] schedule+0x29/0x70 [Fri Apr 12 11:11:23 2019][2937605.020394] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:23 2019][2937605.027719] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:23 2019][2937605.033492] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.040818] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:23 2019][2937605.047387] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.054638] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.061885] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.069225] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.076987] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.084260] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.092024] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.099711] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:23 2019][2937605.106195] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:23 2019][2937605.112225] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:23 2019][2937605.118784] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.125617] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:23 2019][2937605.132075] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:23 2019][2937605.137770] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:23 2019][2937605.145344] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:23 2019][2937605.150422] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.156720] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:23 2019][2937605.163343] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.169629] osp-pre-47-0 S ffff8b7cfc305140 0 117787 2 0x00000080 [Fri Apr 12 11:11:23 2019][2937605.177039] Call Trace: [Fri Apr 12 11:11:23 2019][2937605.179676] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:23 2019][2937605.185078] [] schedule+0x29/0x70 [Fri Apr 12 11:11:23 2019][2937605.190236] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:23 2019][2937605.197205] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:23 2019][2937605.202963] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:23 2019][2937605.209666] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:23 2019][2937605.214727] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.221002] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:23 2019][2937605.227625] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.233899] osp-syn-38-0 S ffff8b7cfc301040 0 117788 2 0x00000080 [Fri Apr 12 11:11:23 2019][2937605.241271] Call Trace: [Fri Apr 12 11:11:23 2019][2937605.243902] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:23 2019][2937605.250002] [] schedule+0x29/0x70 [Fri Apr 12 11:11:23 2019][2937605.255158] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:23 2019][2937605.262474] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:23 2019][2937605.268260] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.275575] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:23 2019][2937605.282123] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.289370] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.296614] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.303947] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.311712] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.318958] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.326724] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.334383] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:23 2019][2937605.340831] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:23 2019][2937605.346852] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:23 2019][2937605.353402] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.360217] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:23 2019][2937605.366667] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:23 2019][2937605.372340] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:23 2019][2937605.379908] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:23 2019][2937605.384970] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.391246] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:23 2019][2937605.397867] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.404151] osp-pre-38-0 S ffff8b7cfc300000 0 117789 2 0x00000080 [Fri Apr 12 11:11:23 2019][2937605.411513] Call Trace: [Fri Apr 12 11:11:23 2019][2937605.414144] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:23 2019][2937605.419552] [] schedule+0x29/0x70 [Fri Apr 12 11:11:23 2019][2937605.424704] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:23 2019][2937605.431672] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:23 2019][2937605.437427] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:23 2019][2937605.444154] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:23 2019][2937605.449210] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.455486] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:23 2019][2937605.462106] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.468380] osp-syn-15-0 S ffff8b79ccfaa080 0 117790 2 0x00000080 [Fri Apr 12 11:11:23 2019][2937605.475743] Call Trace: [Fri Apr 12 11:11:23 2019][2937605.478376] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:23 2019][2937605.484477] [] schedule+0x29/0x70 [Fri Apr 12 11:11:23 2019][2937605.489629] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:23 2019][2937605.496939] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:23 2019][2937605.502716] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.510031] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:23 2019][2937605.516580] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.523825] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.531088] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.538527] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.546489] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.553746] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.561519] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.569208] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:23 2019][2937605.575688] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:23 2019][2937605.581708] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:23 2019][2937605.588278] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.595091] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:23 2019][2937605.601543] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:23 2019][2937605.607306] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:23 2019][2937605.614889] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:23 2019][2937605.619947] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.626223] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:23 2019][2937605.632846] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.639119] osp-pre-15-0 S ffff8b79ccfac100 0 117791 2 0x00000080 [Fri Apr 12 11:11:23 2019][2937605.646509] Call Trace: [Fri Apr 12 11:11:23 2019][2937605.649153] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:23 2019][2937605.654571] [] schedule+0x29/0x70 [Fri Apr 12 11:11:23 2019][2937605.659725] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:23 2019][2937605.666708] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:23 2019][2937605.672475] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:23 2019][2937605.679241] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:23 2019][2937605.684343] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.690659] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:23 2019][2937605.697295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:23 2019][2937605.703583] osp-syn-29-0 S ffff8b79ccfa9040 0 117792 2 0x00000080 [Fri Apr 12 11:11:23 2019][2937605.710971] Call Trace: [Fri Apr 12 11:11:23 2019][2937605.713605] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:23 2019][2937605.719723] [] schedule+0x29/0x70 [Fri Apr 12 11:11:23 2019][2937605.724886] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:23 2019][2937605.732207] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:23 2019][2937605.738009] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.745356] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:23 2019][2937605.751927] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.759187] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.766456] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.773797] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.781559] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:23 2019][2937605.788809] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937605.796574] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937605.804233] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:24 2019][2937605.810708] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:24 2019][2937605.816754] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:24 2019][2937605.823349] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:24 2019][2937605.830175] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:24 2019][2937605.836621] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:24 2019][2937605.842302] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:24 2019][2937605.849928] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:24 2019][2937605.854995] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937605.861270] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:24 2019][2937605.867890] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937605.874166] osp-pre-29-0 S ffff8b79ccfad140 0 117793 2 0x00000080 [Fri Apr 12 11:11:24 2019][2937605.881537] Call Trace: [Fri Apr 12 11:11:24 2019][2937605.884178] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:24 2019][2937605.889622] [] schedule+0x29/0x70 [Fri Apr 12 11:11:24 2019][2937605.894782] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:24 2019][2937605.901790] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:24 2019][2937605.907581] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:24 2019][2937605.914303] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:24 2019][2937605.919364] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937605.925640] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:24 2019][2937605.932278] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937605.938558] osp-syn-1-0 S ffff8b79ccfa8000 0 117794 2 0x00000080 [Fri Apr 12 11:11:24 2019][2937605.945930] Call Trace: [Fri Apr 12 11:11:24 2019][2937605.948583] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:24 2019][2937605.954720] [] schedule+0x29/0x70 [Fri Apr 12 11:11:24 2019][2937605.959898] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:24 2019][2937605.967214] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:24 2019][2937605.973025] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937605.980366] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:24 2019][2937605.986935] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:24 2019][2937605.994190] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.001439] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.008791] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.016572] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.023822] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.031627] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.039288] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:24 2019][2937606.045733] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:24 2019][2937606.051756] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:24 2019][2937606.058322] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.065117] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:24 2019][2937606.071573] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:24 2019][2937606.077270] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:24 2019][2937606.084868] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:24 2019][2937606.089927] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.096200] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:24 2019][2937606.102822] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.109094] osp-pre-1-0 S ffff8b79ccfae180 0 117795 2 0x00000080 [Fri Apr 12 11:11:24 2019][2937606.116451] Call Trace: [Fri Apr 12 11:11:24 2019][2937606.119081] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:24 2019][2937606.124493] [] schedule+0x29/0x70 [Fri Apr 12 11:11:24 2019][2937606.129653] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:24 2019][2937606.136625] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:24 2019][2937606.142380] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:24 2019][2937606.149116] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:24 2019][2937606.154220] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.160495] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:24 2019][2937606.167132] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.173408] osp-syn-22-0 S ffff8b7cf3d21040 0 117796 2 0x00000080 [Fri Apr 12 11:11:24 2019][2937606.180787] Call Trace: [Fri Apr 12 11:11:24 2019][2937606.183427] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:24 2019][2937606.189531] [] schedule+0x29/0x70 [Fri Apr 12 11:11:24 2019][2937606.194692] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:24 2019][2937606.202008] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:24 2019][2937606.207794] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.215132] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:24 2019][2937606.221694] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.228948] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.236209] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.243540] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.251316] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.258570] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.266343] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.274002] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:24 2019][2937606.280453] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:24 2019][2937606.286474] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:24 2019][2937606.293058] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.299880] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:24 2019][2937606.306360] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:24 2019][2937606.312039] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:24 2019][2937606.319636] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:24 2019][2937606.324704] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.330981] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:24 2019][2937606.337633] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.343917] osp-pre-22-0 S ffff8b7cf3d25140 0 117797 2 0x00000080 [Fri Apr 12 11:11:24 2019][2937606.351314] Call Trace: [Fri Apr 12 11:11:24 2019][2937606.353959] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:24 2019][2937606.359385] [] schedule+0x29/0x70 [Fri Apr 12 11:11:24 2019][2937606.364543] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:24 2019][2937606.371515] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:24 2019][2937606.377273] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:24 2019][2937606.384006] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:24 2019][2937606.389091] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.395367] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:24 2019][2937606.401988] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.408260] osp-syn-3-0 S ffff8b7cf3d22080 0 117798 2 0x00000080 [Fri Apr 12 11:11:24 2019][2937606.415633] Call Trace: [Fri Apr 12 11:11:24 2019][2937606.418271] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:24 2019][2937606.424408] [] schedule+0x29/0x70 [Fri Apr 12 11:11:24 2019][2937606.429564] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:24 2019][2937606.436917] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:24 2019][2937606.442700] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.450053] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:24 2019][2937606.456607] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.463852] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.471100] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.478466] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.486245] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.493520] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.501320] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.508981] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:24 2019][2937606.515427] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:24 2019][2937606.521456] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:24 2019][2937606.528015] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.534819] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:24 2019][2937606.541262] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:24 2019][2937606.546935] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:24 2019][2937606.554503] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:24 2019][2937606.559565] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.565841] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:24 2019][2937606.572461] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.578734] osp-pre-3-0 S ffff8b7cf3d24100 0 117799 2 0x00000080 [Fri Apr 12 11:11:24 2019][2937606.586102] Call Trace: [Fri Apr 12 11:11:24 2019][2937606.588742] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:24 2019][2937606.594151] [] schedule+0x29/0x70 [Fri Apr 12 11:11:24 2019][2937606.599322] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:24 2019][2937606.606301] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:24 2019][2937606.612085] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:24 2019][2937606.618795] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:24 2019][2937606.623860] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.630159] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:24 2019][2937606.636779] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:24 2019][2937606.643054] osp-syn-27-0 S ffff8b7cf3d230c0 0 117800 2 0x00000080 [Fri Apr 12 11:11:24 2019][2937606.650425] Call Trace: [Fri Apr 12 11:11:24 2019][2937606.653057] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:24 2019][2937606.659185] [] schedule+0x29/0x70 [Fri Apr 12 11:11:24 2019][2937606.664366] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:24 2019][2937606.671693] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:24 2019][2937606.677467] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.684783] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:24 2019][2937606.691350] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.698605] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.705867] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.713208] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.720981] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.728228] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.735999] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:24 2019][2937606.743680] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:25 2019][2937606.750166] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:25 2019][2937606.756190] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937606.762745] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.769542] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:25 2019][2937606.775985] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:25 2019][2937606.781657] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:25 2019][2937606.789230] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937606.794289] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937606.800565] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937606.807188] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937606.813460] osp-pre-27-0 S ffff8b7cf3d20000 0 117801 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937606.820823] Call Trace: [Fri Apr 12 11:11:25 2019][2937606.823456] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:25 2019][2937606.828863] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937606.834015] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:25 2019][2937606.840979] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937606.846737] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:25 2019][2937606.853441] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937606.858503] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937606.864777] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937606.871400] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937606.877689] osp-syn-37-0 S ffff8b6f65ebc100 0 117802 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937606.885098] Call Trace: [Fri Apr 12 11:11:25 2019][2937606.887728] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:25 2019][2937606.893872] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937606.899024] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:25 2019][2937606.906337] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937606.912111] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.919428] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937606.925977] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.933224] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.940469] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.947800] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.955564] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.962811] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.970576] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937606.978238] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:25 2019][2937606.984685] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:25 2019][2937606.990704] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937606.997256] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.004051] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:25 2019][2937607.010495] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:25 2019][2937607.016166] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:25 2019][2937607.023739] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937607.028797] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.035072] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937607.041713] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.048003] osp-pre-37-0 S ffff8b6f65ebd140 0 117803 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937607.055365] Call Trace: [Fri Apr 12 11:11:25 2019][2937607.057999] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:25 2019][2937607.063404] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937607.068560] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:25 2019][2937607.075532] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937607.081290] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:25 2019][2937607.087993] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937607.093055] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.099329] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937607.105949] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.112228] osp-syn-36-0 S ffff8b6f65eb8000 0 117804 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937607.119587] Call Trace: [Fri Apr 12 11:11:25 2019][2937607.122220] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:25 2019][2937607.128320] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937607.133474] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:25 2019][2937607.140784] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937607.146559] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.153875] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937607.160426] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.167671] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.174936] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.182289] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.190082] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.197380] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.205241] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.213081] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:25 2019][2937607.219606] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:25 2019][2937607.225645] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937607.232201] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.238997] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:25 2019][2937607.245445] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:25 2019][2937607.251118] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:25 2019][2937607.258685] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937607.263746] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.270025] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937607.276643] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.282918] osp-pre-36-0 S ffff8b6f65ebb0c0 0 117805 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937607.290282] Call Trace: [Fri Apr 12 11:11:25 2019][2937607.292913] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:25 2019][2937607.298321] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937607.303474] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:25 2019][2937607.310436] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937607.316198] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:25 2019][2937607.322908] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937607.327971] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.334245] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937607.340867] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.347139] osp-syn-8-0 S ffff8b6f65ebe180 0 117806 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937607.354538] Call Trace: [Fri Apr 12 11:11:25 2019][2937607.357188] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:25 2019][2937607.363376] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937607.368597] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:25 2019][2937607.375988] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937607.381874] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.389298] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937607.395929] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.403284] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.410549] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.417882] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.425647] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.432891] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.440657] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.448318] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:25 2019][2937607.454780] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:25 2019][2937607.460794] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937607.467344] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.474142] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:25 2019][2937607.480585] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:25 2019][2937607.486257] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:25 2019][2937607.493828] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937607.498888] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.505161] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937607.511777] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.518064] osp-pre-8-0 S ffff8b6f65eba080 0 117807 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937607.525421] Call Trace: [Fri Apr 12 11:11:25 2019][2937607.528054] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:25 2019][2937607.533459] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937607.538604] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:25 2019][2937607.545569] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937607.551326] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:25 2019][2937607.558030] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937607.563092] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.569368] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937607.575987] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.582252] osp-syn-35-0 S ffff8b7cf3005140 0 117808 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937607.589611] Call Trace: [Fri Apr 12 11:11:25 2019][2937607.592241] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:25 2019][2937607.598340] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937607.603494] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:25 2019][2937607.610805] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937607.616579] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.623897] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937607.630446] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.637691] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.644935] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.652269] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.660033] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.667281] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.675044] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.682705] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:25 2019][2937607.689153] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:25 2019][2937607.695169] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:25 2019][2937607.701722] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.708533] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:25 2019][2937607.714982] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:25 2019][2937607.720653] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:25 2019][2937607.728223] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937607.733284] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.739558] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937607.746181] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.752454] osp-pre-35-0 S ffff8b7cf3004100 0 117809 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937607.759818] Call Trace: [Fri Apr 12 11:11:25 2019][2937607.762451] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:25 2019][2937607.767856] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937607.773010] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:25 2019][2937607.779974] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:11:25 2019][2937607.785728] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937607.791486] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:25 2019][2937607.798191] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:25 2019][2937607.803252] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.809526] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:25 2019][2937607.816149] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:25 2019][2937607.822420] osp-syn-28-0 S ffff8b7385bc0000 0 117810 2 0x00000080 [Fri Apr 12 11:11:25 2019][2937607.829793] Call Trace: [Fri Apr 12 11:11:25 2019][2937607.832423] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:25 2019][2937607.838526] [] schedule+0x29/0x70 [Fri Apr 12 11:11:25 2019][2937607.843678] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:25 2019][2937607.850988] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:25 2019][2937607.856764] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:25 2019][2937607.864081] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:26 2019][2937607.870631] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:26 2019][2937607.877878] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:26 2019][2937607.885121] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:26 2019][2937607.892453] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:26 2019][2937607.900221] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:26 2019][2937607.907463] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:26 2019][2937607.915231] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:26 2019][2937607.922890] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:26 2019][2937607.929339] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:26 2019][2937607.935357] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:26 2019][2937607.941906] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:26 2019][2937607.948703] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:26 2019][2937607.955146] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:26 2019][2937607.960835] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:26 2019][2937607.968406] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937607.973467] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937607.979738] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937607.986357] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937607.992629] osp-pre-28-0 S ffff8b7385bc2080 0 117811 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937607.999984] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.002616] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:26 2019][2937608.008022] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.013168] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:26 2019][2937608.020146] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.025898] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:26 2019][2937608.032602] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937608.037663] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.043939] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937608.050559] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.056833] osp-syn-30-0 S ffff8b7385bc5140 0 117812 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937608.064188] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.066821] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:26 2019][2937608.072922] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.078088] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:26 2019][2937608.085404] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.091175] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.098492] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:26 2019][2937608.105042] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.112291] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.119531] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.126865] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.134632] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.141879] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.149644] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.157304] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:26 2019][2937608.163749] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:26 2019][2937608.169769] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:26 2019][2937608.176321] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.183117] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:26 2019][2937608.189562] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:26 2019][2937608.195234] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:26 2019][2937608.202802] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937608.207866] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.214140] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937608.220761] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.227034] osp-pre-30-0 S ffff8b7385bc4100 0 117813 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937608.234407] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.237048] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:26 2019][2937608.242454] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.247610] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:26 2019][2937608.254581] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.260341] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:26 2019][2937608.267094] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937608.272167] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.278441] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937608.285071] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.291342] osp-syn-34-0 S ffff8b7385bc6180 0 117814 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937608.298705] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.301340] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:26 2019][2937608.307439] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.312591] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:11:26 2019][2937608.319900] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.325688] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.333015] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:26 2019][2937608.339580] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.346823] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.354070] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.361420] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.369192] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.376444] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.384214] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.391880] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:26 2019][2937608.398332] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:26 2019][2937608.404370] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:11:26 2019][2937608.410936] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.417767] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:11:26 2019][2937608.424224] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:11:26 2019][2937608.429918] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:11:26 2019][2937608.437522] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937608.442582] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.448878] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937608.455523] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.461806] osp-pre-34-0 S ffff8b7385bc1040 0 117815 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937608.469194] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.471826] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:11:26 2019][2937608.477251] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.482415] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:11:26 2019][2937608.489393] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.495195] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:11:26 2019][2937608.501925] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937608.506986] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.513263] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937608.519882] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.526155] lquota_wb_fir-M S ffff8b79fc4ad140 0 117833 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937608.533518] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.536148] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.541296] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:11:26 2019][2937608.547311] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:11:26 2019][2937608.553854] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Fri Apr 12 11:11:26 2019][2937608.560657] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Fri Apr 12 11:11:26 2019][2937608.567280] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.573040] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Fri Apr 12 11:11:26 2019][2937608.579393] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937608.584448] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.590723] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937608.597355] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.603630] lquota_wb_fir-M S ffff8b79c36b9040 0 117836 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937608.611001] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.613631] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.618787] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:11:26 2019][2937608.624811] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:11:26 2019][2937608.631354] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Fri Apr 12 11:11:26 2019][2937608.638192] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Fri Apr 12 11:11:26 2019][2937608.644818] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.650575] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Fri Apr 12 11:11:26 2019][2937608.656941] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937608.661998] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.668276] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937608.674908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.681179] dist_txn-0 S ffff8b79c36bd140 0 117837 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937608.688570] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.691210] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:26 2019][2937608.697316] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.702531] [] distribute_txn_commit_thread+0x9ac/0xca0 [ptlrpc] [Fri Apr 12 11:11:26 2019][2937608.710372] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.716183] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [Fri Apr 12 11:11:26 2019][2937608.723252] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:26 2019][2937608.728330] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.734603] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:26 2019][2937608.741226] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:26 2019][2937608.747505] mdt00_007 S ffff8b6cfc4f4100 0 117869 2 0x00000080 [Fri Apr 12 11:11:26 2019][2937608.754873] Call Trace: [Fri Apr 12 11:11:26 2019][2937608.757548] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:26 2019][2937608.765295] [] schedule+0x29/0x70 [Fri Apr 12 11:11:26 2019][2937608.770475] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:26 2019][2937608.777582] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:26 2019][2937608.784640] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:26 2019][2937608.790423] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:26 2019][2937608.797654] [] ? kfree+0x106/0x140 [Fri Apr 12 11:11:26 2019][2937608.802921] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:26 2019][2937608.809753] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:26 2019][2937608.817774] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:26 2019][2937608.824398] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:26 2019][2937608.831119] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:26 2019][2937608.837507] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.844564] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:26 2019][2937608.852251] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:26 2019][2937608.859580] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:26 2019][2937608.865867] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:27 2019][2937608.873292] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:27 2019][2937608.880202] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:27 2019][2937608.886607] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937608.893797] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:27 2019][2937608.900795] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:27 2019][2937608.907454] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:27 2019][2937608.915141] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:27 2019][2937608.921431] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:27 2019][2937608.927372] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:27 2019][2937608.933917] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:27 2019][2937608.940161] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:27 2019][2937608.946885] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:27 2019][2937608.953892] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:27 2019][2937608.959806] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937608.966900] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937608.974674] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:27 2019][2937608.981954] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937608.989827] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937608.996793] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:27 2019][2937609.003241] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:27 2019][2937609.009295] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.015775] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.023358] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:27 2019][2937609.028437] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.034722] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:27 2019][2937609.041339] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.047648] mdt01_007 S ffff8b6cfc4f1040 0 117898 2 0x00000080 [Fri Apr 12 11:11:27 2019][2937609.055031] Call Trace: [Fri Apr 12 11:11:27 2019][2937609.057683] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:27 2019][2937609.064560] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:27 2019][2937609.070735] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.078496] [] schedule+0x29/0x70 [Fri Apr 12 11:11:27 2019][2937609.083675] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.090775] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.097870] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:27 2019][2937609.105008] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:27 2019][2937609.110796] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.118144] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.126184] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.133941] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.140415] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.148192] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.156214] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.162860] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:27 2019][2937609.170542] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:27 2019][2937609.176822] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:27 2019][2937609.182756] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:27 2019][2937609.189303] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:27 2019][2937609.195497] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.202215] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.209193] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:27 2019][2937609.215083] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.222180] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.229929] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:27 2019][2937609.237197] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.245065] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.252031] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:27 2019][2937609.258479] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:27 2019][2937609.264533] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.271013] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.278597] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:27 2019][2937609.283671] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:11:27 2019][2937609.289976] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.296261] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:27 2019][2937609.302883] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.309165] ldlm_cn02_006 S ffff8b6cfc4f30c0 0 117901 2 0x00000080 [Fri Apr 12 11:11:27 2019][2937609.316569] Call Trace: [Fri Apr 12 11:11:27 2019][2937609.319212] [] schedule+0x29/0x70 [Fri Apr 12 11:11:27 2019][2937609.324428] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.331309] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:27 2019][2937609.337097] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.343581] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.351153] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:27 2019][2937609.356215] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.362506] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:27 2019][2937609.369120] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.375396] mdt_rdpg01_002 S ffff8b6cfc4f0000 0 117902 2 0x00000080 [Fri Apr 12 11:11:27 2019][2937609.382759] Call Trace: [Fri Apr 12 11:11:27 2019][2937609.385390] [] schedule+0x29/0x70 [Fri Apr 12 11:11:27 2019][2937609.390604] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.397512] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:27 2019][2937609.403348] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.409832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.417407] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:27 2019][2937609.422520] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.428795] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:27 2019][2937609.435434] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.441704] ll_mgs_0005 S ffff8b5f3476e180 0 117903 2 0x00000080 [Fri Apr 12 11:11:27 2019][2937609.449068] Call Trace: [Fri Apr 12 11:11:27 2019][2937609.451698] [] schedule+0x29/0x70 [Fri Apr 12 11:11:27 2019][2937609.456889] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.463773] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:27 2019][2937609.469561] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.476043] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.483620] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:27 2019][2937609.488686] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.495001] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:27 2019][2937609.501622] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.507912] ll_mgs_0006 S ffff8b5f3476b0c0 0 117904 2 0x00000080 [Fri Apr 12 11:11:27 2019][2937609.515273] Call Trace: [Fri Apr 12 11:11:27 2019][2937609.517906] [] schedule+0x29/0x70 [Fri Apr 12 11:11:27 2019][2937609.523101] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.530003] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:27 2019][2937609.535820] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.542324] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.549915] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:27 2019][2937609.554983] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.561266] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:27 2019][2937609.567886] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.574160] ll_mgs_0007 S ffff8b5f3476a080 0 117905 2 0x00000080 [Fri Apr 12 11:11:27 2019][2937609.581523] Call Trace: [Fri Apr 12 11:11:27 2019][2937609.584154] [] schedule+0x29/0x70 [Fri Apr 12 11:11:27 2019][2937609.589343] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.596256] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:27 2019][2937609.602047] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.608528] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.616100] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:27 2019][2937609.621162] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:11:27 2019][2937609.627439] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.633741] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:27 2019][2937609.640360] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:27 2019][2937609.646633] mdt01_008 S ffff8b5f3476d140 0 117906 2 0x00000080 [Fri Apr 12 11:11:27 2019][2937609.654007] Call Trace: [Fri Apr 12 11:11:27 2019][2937609.656686] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.664483] [] schedule+0x29/0x70 [Fri Apr 12 11:11:27 2019][2937609.669670] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.676756] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.683836] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:27 2019][2937609.689623] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.696877] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:27 2019][2937609.704222] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.711043] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.719066] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:27 2019][2937609.725691] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:27 2019][2937609.732417] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:27 2019][2937609.738822] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:27 2019][2937609.745897] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:27 2019][2937609.753584] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:27 2019][2937609.760915] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:27 2019][2937609.767202] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:27 2019][2937609.774610] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:27 2019][2937609.781502] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.787906] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.795055] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.802041] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.808703] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:27 2019][2937609.816412] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:27 2019][2937609.822716] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:27 2019][2937609.828667] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:27 2019][2937609.835212] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:27 2019][2937609.841417] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.848134] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:27 2019][2937609.855108] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:27 2019][2937609.861003] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:27 2019][2937609.868114] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937609.875889] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:28 2019][2937609.883177] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937609.891051] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937609.898018] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:28 2019][2937609.904467] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:28 2019][2937609.910521] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937609.917013] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937609.924612] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:28 2019][2937609.929676] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937609.935953] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:28 2019][2937609.942574] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937609.948848] mdt_rdpg01_003 S ffff8b5f34768000 0 117908 2 0x00000080 [Fri Apr 12 11:11:28 2019][2937609.956211] Call Trace: [Fri Apr 12 11:11:28 2019][2937609.958862] [] schedule+0x29/0x70 [Fri Apr 12 11:11:28 2019][2937609.964050] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937609.970934] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:28 2019][2937609.976723] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937609.983208] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937609.990825] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:28 2019][2937609.995939] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.002238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:28 2019][2937610.008864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.015138] mdt_rdpg01_004 S ffff8b5f34769040 0 117909 2 0x00000080 [Fri Apr 12 11:11:28 2019][2937610.022537] Call Trace: [Fri Apr 12 11:11:28 2019][2937610.025167] [] schedule+0x29/0x70 [Fri Apr 12 11:11:28 2019][2937610.030360] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.037248] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:28 2019][2937610.043046] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.049541] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.057126] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:28 2019][2937610.062205] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.068495] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:28 2019][2937610.075136] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.081406] mdt02_009 S ffff8b774877c100 0 117911 2 0x00000080 [Fri Apr 12 11:11:28 2019][2937610.088813] Call Trace: [Fri Apr 12 11:11:28 2019][2937610.091452] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:28 2019][2937610.098371] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.106127] [] schedule+0x29/0x70 [Fri Apr 12 11:11:28 2019][2937610.111335] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.118430] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.125542] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:28 2019][2937610.132692] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:28 2019][2937610.138498] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.145842] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.153855] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.160574] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:28 2019][2937610.167549] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.174363] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.182433] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.189561] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.196629] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.203809] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.210444] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:28 2019][2937610.217644] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.224322] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.231419] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.238565] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:28 2019][2937610.245285] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:28 2019][2937610.251915] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.258738] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.265629] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:28 2019][2937610.272951] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:28 2019][2937610.279518] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.286782] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.294485] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.300797] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.307885] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.315635] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:28 2019][2937610.322893] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.330764] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.337730] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:28 2019][2937610.344179] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:28 2019][2937610.350228] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.356730] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.364304] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:28 2019][2937610.369362] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.375638] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:28 2019][2937610.382261] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.388534] mdt03_007 S ffff8b774877e180 0 117968 2 0x00000080 [Fri Apr 12 11:11:28 2019][2937610.395897] Call Trace: [Fri Apr 12 11:11:28 2019][2937610.398563] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.406310] [] schedule+0x29/0x70 [Fri Apr 12 11:11:28 2019][2937610.411490] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.418579] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.425631] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:28 2019][2937610.431448] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.438695] [] ? kfree+0x106/0x140 [Fri Apr 12 11:11:28 2019][2937610.443997] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.450859] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.458914] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:28 2019][2937610.465592] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:28 2019][2937610.472313] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:28 2019][2937610.478719] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:28 2019][2937610.485784] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:28 2019][2937610.493470] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:28 2019][2937610.500795] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:28 2019][2937610.507085] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:28 2019][2937610.514510] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:28 2019][2937610.521420] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.527885] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.535076] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.542082] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.548732] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:28 2019][2937610.556425] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:28 2019][2937610.562719] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:28 2019][2937610.568689] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:28 2019][2937610.575289] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:28 2019][2937610.581506] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.588223] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:28 2019][2937610.595201] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:28 2019][2937610.601099] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.608196] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.615952] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:28 2019][2937610.623222] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.631099] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.638065] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:28 2019][2937610.644510] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:28 2019][2937610.650562] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.657046] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.664618] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:28 2019][2937610.669678] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.675954] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:28 2019][2937610.682576] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.688849] mdt00_008 S ffff8b774877a080 0 117969 2 0x00000080 [Fri Apr 12 11:11:28 2019][2937610.696213] Call Trace: [Fri Apr 12 11:11:28 2019][2937610.698844] [] schedule+0x29/0x70 [Fri Apr 12 11:11:28 2019][2937610.704032] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.710910] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:28 2019][2937610.716701] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.723184] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.730758] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:28 2019][2937610.735815] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.742091] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:28 2019][2937610.748716] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.755005] mdt_rdpg00_002 S ffff8b7748778000 0 117970 2 0x00000080 [Fri Apr 12 11:11:28 2019][2937610.762377] Call Trace: [Fri Apr 12 11:11:28 2019][2937610.765018] [] schedule+0x29/0x70 [Fri Apr 12 11:11:28 2019][2937610.770210] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.777089] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:28 2019][2937610.782893] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.789393] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.796973] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:28 2019][2937610.802042] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.808326] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:28 2019][2937610.814957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:28 2019][2937610.821227] mdt_rdpg01_005 S ffff8b76fa3e2080 0 118026 2 0x00000080 [Fri Apr 12 11:11:28 2019][2937610.828591] Call Trace: [Fri Apr 12 11:11:28 2019][2937610.831222] [] schedule+0x29/0x70 [Fri Apr 12 11:11:28 2019][2937610.836414] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.843298] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:28 2019][2937610.849087] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:28 2019][2937610.855570] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937610.863143] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937610.868201] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937610.874480] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937610.881098] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937610.887370] mdt_rdpg02_002 S ffff8b76fa3e6180 0 118027 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937610.894726] Call Trace: [Fri Apr 12 11:11:29 2019][2937610.897359] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937610.902548] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937610.909430] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937610.915221] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937610.921702] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937610.929278] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937610.934337] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937610.940606] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937610.947225] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937610.953500] mdt_rdpg02_003 S ffff8b76fa3e5140 0 118028 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937610.960863] Call Trace: [Fri Apr 12 11:11:29 2019][2937610.963497] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937610.968684] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937610.975568] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937610.981359] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937610.987843] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937610.995417] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.000474] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.006741] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.013364] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.019636] mdt_rdpg02_004 S ffff8b76fa3e30c0 0 118029 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.027009] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.029641] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.034829] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.041713] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.047505] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.053987] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.061559] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.066618] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.072888] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.079510] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.085783] mdt_rdpg03_002 S ffff8b76fa3e0000 0 118082 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.093146] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.095777] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.100966] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.107850] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.113644] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.120126] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.127698] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.132756] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.139027] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.145644] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.151919] mdt_rdpg03_003 S ffff8b7cf30330c0 0 118083 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.159274] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.161905] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.167094] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.173970] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.179759] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.186242] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.193814] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.198877] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.205150] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.211772] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.218046] mdt_rdpg03_004 S ffff8b7cf3035140 0 118084 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.225403] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.228033] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.233219] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.240097] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.245887] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.252372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.259944] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.265005] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.271282] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.277900] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.284166] mdt_rdpg02_005 S ffff8b7cf3032080 0 118085 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.291530] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.294162] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.299348] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.306225] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.312019] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.318500] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.326087] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.331142] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.337446] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.344077] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.350352] mdt_rdpg00_003 S ffff8b7cf3030000 0 118086 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.357721] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.360355] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.365554] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.372443] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.378276] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.384792] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.392385] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.397453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.403728] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.410352] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.416624] mdt_rdpg00_004 S ffff8b7cf3034100 0 118087 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.424005] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.426652] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.431860] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.438746] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.444542] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.451096] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.458671] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.463728] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.470024] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.476646] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.482952] mdt_out01_002 S ffff8b750a6c5140 0 118140 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.490341] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.492978] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.498201] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.505109] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.510938] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.517421] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.525000] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.530057] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.536333] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.542963] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.549275] mdt01_009 S ffff8b8aeca06180 0 118148 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.556651] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.559295] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:29 2019][2937611.566247] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.574037] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.579242] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.586336] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.593422] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:29 2019][2937611.600564] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.606353] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.613726] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.621776] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.629555] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:29 2019][2937611.636493] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.644293] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.652359] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:29 2019][2937611.659008] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:29 2019][2937611.665210] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:29 2019][2937611.671927] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:29 2019][2937611.678918] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:29 2019][2937611.684843] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.691970] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.699726] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:29 2019][2937611.707005] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.714869] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.721837] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:29 2019][2937611.728288] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:29 2019][2937611.734347] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.740842] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.748441] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:29 2019][2937611.753517] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.759809] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:29 2019][2937611.766430] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:29 2019][2937611.772701] mdt01_010 S ffff8b8aeca05140 0 118149 2 0x00000080 [Fri Apr 12 11:11:29 2019][2937611.780066] Call Trace: [Fri Apr 12 11:11:29 2019][2937611.782706] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:29 2019][2937611.789620] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.797368] [] schedule+0x29/0x70 [Fri Apr 12 11:11:29 2019][2937611.802549] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.809639] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.816722] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:29 2019][2937611.823864] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:29 2019][2937611.829653] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.837019] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:29 2019][2937611.845037] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:29 2019][2937611.851756] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:29 2019][2937611.858731] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:30 2019][2937611.865471] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.873521] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.880622] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.887689] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:30 2019][2937611.894839] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:30 2019][2937611.901468] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:30 2019][2937611.908648] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.915306] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.922397] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.929549] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:30 2019][2937611.936266] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:30 2019][2937611.942896] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:30 2019][2937611.949721] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.956613] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:30 2019][2937611.963934] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:30 2019][2937611.970507] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.977773] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.985476] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.991791] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937611.998887] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.006636] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:30 2019][2937612.013896] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.021763] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.028730] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:30 2019][2937612.035180] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:30 2019][2937612.041231] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.047712] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.055290] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:30 2019][2937612.060345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:30 2019][2937612.066620] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:30 2019][2937612.073242] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:30 2019][2937612.079517] mdt01_011 S ffff8b8aeca02080 0 118151 2 0x00000080 [Fri Apr 12 11:11:30 2019][2937612.086880] Call Trace: [Fri Apr 12 11:11:30 2019][2937612.089513] [] schedule+0x29/0x70 [Fri Apr 12 11:11:30 2019][2937612.094698] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.101576] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:30 2019][2937612.107369] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.113850] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.121421] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:30 2019][2937612.126482] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:30 2019][2937612.132759] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:30 2019][2937612.139379] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:30 2019][2937612.145651] mdt01_012 S ffff8b8aeca01040 0 118152 2 0x00000080 [Fri Apr 12 11:11:30 2019][2937612.153007] Call Trace: [Fri Apr 12 11:11:30 2019][2937612.155678] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.163421] [] schedule+0x29/0x70 [Fri Apr 12 11:11:30 2019][2937612.168605] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.175692] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.182741] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:30 2019][2937612.188529] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.195759] [] ? kfree+0x106/0x140 [Fri Apr 12 11:11:30 2019][2937612.201026] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.207857] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.215874] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:30 2019][2937612.222502] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:30 2019][2937612.229220] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:30 2019][2937612eycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:30 2019][2937612.242670] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:30 2019][2937612.250359] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:30 2019][2937612.257681] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:30 2019][2937612.263972] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:30 2019][2937612.271380] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:30 2019][2937612.278272] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.284670] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.291824] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.298800] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.305450] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:30 2019][2937612.313133] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:30 2019][2937612.319419] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:30 2019][2937612.325354] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:30 2019][2937612.331901] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:30 2019][2937612.338094] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.344813] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.351791] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:30 2019][2937612.357680] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.364777] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.372533] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:30 2019][2937612.379792] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.387661] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.394629] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:30 2019][2937612.401076] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:30 2019][2937612.407129] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.413611] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.421183] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:30 2019][2937612.426243] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:30 2019][2937612.432519] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:30 2019][2937612.439141] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:30 2019][2937612.445414] mdt00_009 S ffff8b8aeca030c0 0 118153 2 0x00000080 [Fri Apr 12 11:11:30 2019][2937612.452779] Call Trace: [Fri Apr 12 11:11:30 2019][2937612.455413] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:30 2019][2937612.462338] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.470082] [] schedule+0x29/0x70 [Fri Apr 12 11:11:30 2019][2937612.475263] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.482350] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.489427] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:30 2019][2937612.496568] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:30 2019][2937612.502356] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.509707] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.517747] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.525514] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:30 2019][2937612.532429] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.540212] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.548234] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:30 2019][2937612.554864] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:30 2019][2937612.561059] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.567779] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.574752] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:30 2019][2937612.580642] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.587728] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.595479] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:30 2019][2937612.602740] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.610608] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.617575] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:30 2019][2937612.624024] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:30 2019][2937612.630078] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.636566] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.644138] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:30 2019][2937612.649199] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:30 2019][2937612.655476] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:30 2019][2937612.662094] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:30 2019][2937612.668368] mdt01_013 S ffff8b8aeca04100 0 118155 2 0x00000080 [Fri Apr 12 11:11:30 2019][2937612.675723] Call Trace: [Fri Apr 12 11:11:30 2019][2937612.678361] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:30 2019][2937612.685281] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.693029] [] schedule+0x29/0x70 [Fri Apr 12 11:11:30 2019][2937612.698212] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.705300] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.712379] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:30 2019][2937612.719513] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:30 2019][2937612.725311] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.732661] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.740682] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.747397] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:30 2019][2937612.754375] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.761114] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.769164] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.776254] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.783323] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.790473] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.797102] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:30 2019][2937612.804282] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.810935] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.818022] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.825173] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:30 2019][2937612.831891] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:30 2019][2937612.838522] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:30 2019][2937612.845346] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:30 2019][2937612.852237] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:30 2019][2937612.859560] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:31 2019][2937612.866131] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.873399] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.881101] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.887414] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.894502] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.902254] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:31 2019][2937612.909522] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.917388] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.924358] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:31 2019][2937612.930802] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:31 2019][2937612.936854] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.943339] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937612.950911] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:31 2019][2937612.955972] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:11:31 2019][2937612.962248] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937612.968525] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:31 2019][2937612.975145] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937612.981434] mdt01_014 S ffff8b6cf7ef4100 0 118156 2 0x00000080 [Fri Apr 12 11:11:31 2019][2937612.988798] Call Trace: [Fri Apr 12 11:11:31 2019][2937612.991438] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:31 2019][2937612.998355] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.006102] [] schedule+0x29/0x70 [Fri Apr 12 11:11:31 2019][2937613.011284] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.018373] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.025453] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:31 2019][2937613.032598] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:31 2019][2937613.038384] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.045736] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.053754] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.060473] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:31 2019][2937613.067448] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.074188] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.082208] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.089390] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.096544] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:11:31 2019][2937613.103720] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.110378] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.117479] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.124631] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:31 2019][2937613.131349] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:31 2019][2937613.137978] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.144803] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.151689] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:31 2019][2937613.159008] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:31 2019][2937613.165572] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.172835] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.180540] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.186857] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.193967] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.201717] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:31 2019][2937613.208978] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.216845] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.223816] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:31 2019][2937613.230260] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:31 2019][2937613.236316] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.242800] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.250378] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:31 2019][2937613.255436] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.261713] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:31 2019][2937613.268334] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.274608] mdt00_010 S ffff8b6cf7ef5140 0 118157 2 0x00000080 [Fri Apr 12 11:11:31 2019][2937613.281971] Call Trace: [Fri Apr 12 11:11:31 2019][2937613.284607] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:31 2019][2937613.291484] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:31 2019][2937613.297618] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.305368] [] schedule+0x29/0x70 [Fri Apr 12 11:11:31 2019][2937613.310548] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.317643] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.324723] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:31 2019][2937613.331861] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:31 2019][2937613.337648] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.344991] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.353036] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.360797] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.367283] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.375074] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.383096] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.389746] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:31 2019][2937613.397426] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:31 2019][2937613.403713] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:31 2019][2937613.409650] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:31 2019][2937613.416194] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:31 2019][2937613.422391] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.429109] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.436082] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:31 2019][2937613.441974] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.449074] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.456826] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:31 2019][2937613.464091] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.471964] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.478931] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:31 2019][2937613.485378] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:31 2019][2937613.491432] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.497912] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.505485] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:31 2019][2937613.510547] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.516822] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:31 2019][2937613.523442] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.529719] mdt00_011 S ffff8b6cf7ef30c0 0 118158 2 0x00000080 [Fri Apr 12 11:11:31 2019][2937613.537080] Call Trace: [Fri Apr 12 11:11:31 2019][2937613.539711] [] schedule+0x29/0x70 [Fri Apr 12 11:11:31 2019][2937613.544900] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.551776] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:31 2019][2937613.557565] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.564051] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.571622] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:31 2019][2937613.576684] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.582961] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:31 2019][2937613.589579] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.595854] mdt00_012 S ffff8b7966a98000 0 118160 2 0x00000080 [Fri Apr 12 11:11:31 2019][2937613.603218] Call Trace: [Fri Apr 12 11:11:31 2019][2937613.605848] [] schedule+0x29/0x70 [Fri Apr 12 11:11:31 2019][2937613.611035] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.617913] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:31 2019][2937613.623706] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.630187] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.637760] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:31 2019][2937613.642837] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.649113] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:31 2019][2937613.655733] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.662006] mdt02_010 S ffff8b7966a99040 0 118161 2 0x00000080 [Fri Apr 12 11:11:31 2019][2937613.669371] Call Trace: [Fri Apr 12 11:11:31 2019][2937613.672000] [] schedule+0x29/0x70 [Fri Apr 12 11:11:31 2019][2937613.677184] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.684065] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:31 2019][2937613.689859] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.696342] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.703925] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:31 2019][2937613.708984] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.715259] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:31 2019][2937613.721878] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:31 2019][2937613.728153] mdt01_015 S ffff8b7966a9a080 0 118163 2 0x00000080 [Fri Apr 12 11:11:31 2019][2937613.735509] Call Trace: [Fri Apr 12 11:11:31 2019][2937613.738148] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:31 2019][2937613.745066] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.752813] [] schedule+0x29/0x70 [Fri Apr 12 11:11:31 2019][2937613.757996] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.765079] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.772155] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:31 2019][2937613.779298] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:31 2019][2937613.785088] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.792436] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.800455] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.807173] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:31 2019][2937613.814150] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.820890] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.828940] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.836038] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:31 2019][2937613.843096] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.850248] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:31 2019][2937613.856877] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:31 2019][2937613.864055] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.870709] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.877797] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.884949] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:32 2019][2937613.891667] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:32 2019][2937613.898298] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:32 2019][2937613.905122] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.912012] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:32 2019][2937613.919335] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:32 2019][2937613.925906] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.933174] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.940877] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.947192] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.954287] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.962037] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:32 2019][2937613.969296] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.977163] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937613.984132] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:32 2019][2937613.990578] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:32 2019][2937613.996636] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.003125] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.010696] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:32 2019][2937614.015756] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.022030] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:32 2019][2937614.028651] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.034925] mdt03_008 S ffff8b7966a9c100 0 118166 2 0x00000080 [Fri Apr 12 11:11:32 2019][2937614.042282] Call Trace: [Fri Apr 12 11:11:32 2019][2937614.044910] [] schedule+0x29/0x70 [Fri Apr 12 11:11:32 2019][2937614.050099] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.056975] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:32 2019][2937614.062769] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.069251] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.076823] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:32 2019][2937614.081883] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.088160] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:32 2019][2937614.094778] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.101043] mdt00_013 S ffff8b7966a9e180 0 118167 2 0x00000080 [Fri Apr 12 11:11:32 2019][2937614.108401] Call Trace: [Fri Apr 12 11:11:32 2019][2937614.111035] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:32 2019][2937614.117950] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.125695] [] schedule+0x29/0x70 [Fri Apr 12 11:11:32 2019][2937614.130875] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.137955] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.145035] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:32 2019][2937614.152172] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:32 2019][2937614.157963] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.165310] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.173332] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.180049] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:32 2019][2937614.187022] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.193764] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.201813] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.208903] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.215971] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.223121] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.229751] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:32 2019][2937614.236947] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.243601] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.250690] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.257841] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:32 2019][2937614.264559] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:32 2019][2937614.271188] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.278013] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.284895] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:32 2019][2937614.292217] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:32 2019][2937614.298783] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.306045] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.313748] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.320064] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.327157] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.334910] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:32 2019][2937614.342168] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.350035] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.357006] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:32 2019][2937614.363453] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:32 2019][2937614.369504] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.376002] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.383578] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:32 2019][2937614.388638] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.394917] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:32 2019][2937614.401535] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.407810] mdt00_014 S ffff8b7966a9d140 0 118168 2 0x00000080 [Fri Apr 12 11:11:32 2019][2937614.415171] Call Trace: [Fri Apr 12 11:11:32 2019][2937614.417808] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:32 2019][2937614.424728] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.432476] [] schedule+0x29/0x70 [Fri Apr 12 11:11:32 2019][2937614.437661] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.444745] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.451821] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:32 2019][2937614.458962] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:32 2019][2937614.464751] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.472098] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.480132] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.487893] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:32 2019][2937614.494807] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.502588] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.510603] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:32 2019][2937614.517234] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:32 2019][2937614.523428] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.530146] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.537126] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:32 2019][2937614.543010] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.550109] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.557856] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:32 2019][2937614.565120] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.572986] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.579953] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:32 2019][2937614.586400] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:32 2019][2937614.592454] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.598933] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.606506] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:32 2019][2937614.611567] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.617846] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:32 2019][2937614.624464] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.630738] mdt01_016 S ffff8b7966a9b0c0 0 118169 2 0x00000080 [Fri Apr 12 11:11:32 2019][2937614.638101] Call Trace: [Fri Apr 12 11:11:32 2019][2937614.640732] [] schedule+0x29/0x70 [Fri Apr 12 11:11:32 2019][2937614.645920] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.652797] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:32 2019][2937614.658589] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.665071] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.672643] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:32 2019][2937614.677705] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.683980] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:32 2019][2937614.690601] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:32 2019][2937614.696875] mdt01_017 S ffff8b7cfede5140 0 118170 2 0x00000080 [Fri Apr 12 11:11:32 2019][2937614.704255] Call Trace: [Fri Apr 12 11:11:32 2019][2937614.706892] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:32 2019][2937614.713812] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.721560] [] schedule+0x29/0x70 [Fri Apr 12 11:11:32 2019][2937614.726745] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.733834] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.740912] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:32 2019][2937614.748054] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:32 2019][2937614.753843] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.761195] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.769238] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.777004] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:32 2019][2937614.783915] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.791691] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.799713] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:32 2019][2937614.806345] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:32 2019][2937614.812546] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.819266] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:32 2019][2937614.826240] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:32 2019][2937614.832128] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.839217] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.846965] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:32 2019][2937614.854225] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.862093] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:32 2019][2937614.869062] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:33 2019][2937614.875509] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:33 2019][2937614.881562] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937614.888042] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937614.895617] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:33 2019][2937614.900677] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937614.906954] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:33 2019][2937614.913572] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937614.919848] mdt02_011 S ffff8b7cfede1040 0 118171 2 0x00000080 [Fri Apr 12 11:11:33 2019][2937614.927210] Call Trace: [Fri Apr 12 11:11:33 2019][2937614.929850] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:33 2019][2937614.936767] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937614.944514] [] schedule+0x29/0x70 [Fri Apr 12 11:11:33 2019][2937614.949697] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937614.956785] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937614.963869] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:33 2019][2937614.971009] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:33 2019][2937614.976796] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937614.984136] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937614.992148] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:33 2019][2937614.998867] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:33 2019][2937615.005844] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.012584] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.020631] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.027720] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.034783] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.041948] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.048581] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:33 2019][2937615.055759] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.062415] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.069504] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.076654] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:33 2019][2937615.083370] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:33 2019][2937615.089999] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.096822] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.103706] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:33 2019][2937615.111028] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:33 2019][2937615.117602] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.124866] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.132571] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.138885] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.145981] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.153740] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:33 2019][2937615.161006] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.168877] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.175844] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:33 2019][2937615.181637] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.188117] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.195689] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:33 2019][2937615.200749] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.207026] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:33 2019][2937615.213647] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.219920] mdt00_015 S ffff8b7cfede30c0 0 118172 2 0x00000080 [Fri Apr 12 11:11:33 2019][2937615.227284] Call Trace: [Fri Apr 12 11:11:33 2019][2937615.229916] [] schedule+0x29/0x70 [Fri Apr 12 11:11:33 2019][2937615.235103] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.241979] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:33 2019][2937615.247769] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.254255] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.261825] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:33 2019][2937615.266887] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.273162] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:33 2019][2937615.279783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.286056] mdt01_018 S ffff8b7cfede6180 0 118173 2 0x00000080 [Fri Apr 12 11:11:33 2019][2937615.293412] Call Trace: [Fri Apr 12 11:11:33 2019][2937615.296043] [] schedule+0x29/0x70 [Fri Apr 12 11:11:33 2019][2937615.301231] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.308106] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:33 2019][2937615.313901] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.320386] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.327961] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:33 2019][2937615.333022] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.339289] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:33 2019][2937615.345910] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.352185] mdt02_012 S ffff8b7cfede4100 0 118174 2 0x00000080 [Fri Apr 12 11:11:33 2019][2937615.359541] Call Trace: [Fri Apr 12 11:11:33 2019][2937615.362177] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:33 2019][2937615.369099] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.376842] [] schedule+0x29/0x70 [Fri Apr 12 11:11:33 2019][2937615.382025] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.389115] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.396198] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:33 2019][2937615.403337] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:33 2019][2937615.409126] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.416478] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.424520] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.432288] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:33 2019][2937615.439201] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.446984] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.455004] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:33 2019][2937615.461637] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:33 2019][2937615.467830] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.474550] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.481523] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:33 2019][2937615.487414] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.494513] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.502267] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:33 2019][2937615.509528] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.517396] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.524364] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:33 2019][2937615.530811] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:33 2019][2937615.536863] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.543345] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.550919] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:33 2019][2937615.555979] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.562255] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:33 2019][2937615.568875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.575149] mdt00_016 S ffff8b7cfede0000 0 118175 2 0x00000080 [Fri Apr 12 11:11:33 2019][2937615.582503] Call Trace: [Fri Apr 12 11:11:33 2019][2937615.585136] [] schedule+0x29/0x70 [Fri Apr 12 11:11:33 2019][2937615.590327] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.597209] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:33 2019][2937615.602998] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.609481] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.617054] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:33 2019][2937615.622116] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.628391] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:33 2019][2937615.635010] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:33 2019][2937615.641276] mdt01_019 S ffff8b7cfe58d140 0 118176 2 0x00000080 [Fri Apr 12 11:11:33 2019][2937615.648632] Call Trace: [Fri Apr 12 11:11:33 2019][2937615.651302] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.659046] [] schedule+0x29/0x70 [Fri Apr 12 11:11:33 2019][2937615.664227] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.671317] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.678372] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:33 2019][2937615.684163] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.691431] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:33 2019][2937615.698773] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.705601] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.713613] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:33 2019][2937615.720243] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:33 2019][2937615.726960] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:33 2019][2937615.733348] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:33 2019][2937615.740408] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:33 2019][2937615.748093] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:33 2019][2937615.755413] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:33 2019][2937615.761700] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:33 2019][2937615.769110] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:33 2019][2937615.776001] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.782405] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.789555] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.796535] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.803189] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:33 2019][2937615.810871] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:33 2019][2937615.817158] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:33 2019][2937615.823096] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:33 2019][2937615.829640] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:33 2019][2937615.835835] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.842552] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:33 2019][2937615.849530] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:33 2019][2937615.855420] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.862516] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:33 2019][2937615.870273] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:34 2019][2937615.877544] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937615.885413] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937615.892376] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:34 2019][2937615.898821] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:34 2019][2937615.904876] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937615.911358] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937615.918937] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:34 2019][2937615.924001] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937615.930275] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:34 2019][2937615.936897] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937615.943169] mdt00_017 S ffff8b7cfe58e180 0 118177 2 0x00000080 [Fri Apr 12 11:11:34 2019][2937615.950526] Call Trace: [Fri Apr 12 11:11:34 2019][2937615.953162] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:34 2019][2937615.960082] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937615.967830] [] schedule+0x29/0x70 [Fri Apr 12 11:11:34 2019][2937615.973012] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937615.980097] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937615.987174] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:34 2019][2937615.994314] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:34 2019][2937616.000102] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.007451] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.015473] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.022190] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:34 2019][2937616.029183] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.035924] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.043971] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.051064] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.058132] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.065283] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.071915] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:34 2019][2937616.079092] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.085747] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.092834] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.099983] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:34 2019][2937616.106703] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:34 2019][2937616.113331] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.120155] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.127039] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:34 2019][2937616.134363] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:34 2019][2937616.140933] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.148200] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.155901] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.162215] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.169302] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.177053] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:34 2019][2937616.184310] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.192179] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.199147] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:34 2019][2937616.205594] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:34 2019][2937616.211647] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.218131] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.225704] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:34 2019][2937616.230761] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.237032] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:34 2019][2937616.243651] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.249926] mdt02_013 S ffff8b7cfe588000 0 118178 2 0x00000080 [Fri Apr 12 11:11:34 2019][2937616.257290] Call Trace: [Fri Apr 12 11:11:34 2019][2937616.259925] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:34 2019][2937616.266847] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.274592] [] schedule+0x29/0x70 [Fri Apr 12 11:11:34 2019][2937616.279779] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.286862] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.293936] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:34 2019][2937616.301079] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:34 2019][2937616.306867] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.314216] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.322236] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.328954] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:34 2019][2937616.335931] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.342668] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.350711] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.357800] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.364860] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.372012] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.378641] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:34 2019][2937616.385818] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.392476] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.399562] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.406711] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:34 2019][2937616.413429] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:34 2019][2937616.420059] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.426891] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.433786] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:34 2019][2937616.441105] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:34 2019][2937616.447670] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.454937] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.462642] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.468955] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.476050] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.483797] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:34 2019][2937616.491059] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.498930] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.505896] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:34 2019][2937616.512358] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:34 2019][2937616.518417] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.524904] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.532476] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:34 2019][2937616.537537] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.543811] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:34 2019][2937616.550432] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.556705] mdt03_009 S ffff8b7cfe589040 0 118179 2 0x00000080 [Fri Apr 12 11:11:34 2019][2937616.564061] Call Trace: [Fri Apr 12 11:11:34 2019][2937616.566692] [] schedule+0x29/0x70 [Fri Apr 12 11:11:34 2019][2937616.571880] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.578756] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:34 2019][2937616.584550] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.591031] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.598603] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:34 2019][2937616.603664] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.609940] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:34 2019][2937616.616560] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.622835] mdt00_018 S ffff8b7cfe58a080 0 118181 2 0x00000080 [Fri Apr 12 11:11:34 2019][2937616.630198] Call Trace: [Fri Apr 12 11:11:34 2019][2937616.632831] [] schedule+0x29/0x70 [Fri Apr 12 11:11:34 2019][2937616.638020] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.644902] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:34 2019][2937616.650694] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.657175] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.664748] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:34 2019][2937616.669810] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.676085] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:34 2019][2937616.682706] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.688980] mdt01_020 S ffff8b7cfe58c100 0 118182 2 0x00000080 [Fri Apr 12 11:11:34 2019][2937616.696343] Call Trace: [Fri Apr 12 11:11:34 2019][2937616.698975] [] schedule+0x29/0x70 [Fri Apr 12 11:11:34 2019][2937616.704165] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.711048] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:34 2019][2937616.716842] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.723323] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.730893] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:34 2019][2937616.735954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.742233] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:34 2019][2937616.748850] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:34 2019][2937616.755125] mdt00_019 S ffff8b6da9842080 0 118183 2 0x00000080 [Fri Apr 12 11:11:34 2019][2937616.762490] Call Trace: [Fri Apr 12 11:11:34 2019][2937616.765127] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:34 2019][2937616.772010] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:34 2019][2937616.778146] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.785895] [] schedule+0x29/0x70 [Fri Apr 12 11:11:34 2019][2937616.791075] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.798163] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.805247] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:34 2019][2937616.812390] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:34 2019][2937616.818193] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.825535] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.833579] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:34 2019][2937616.841341] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:34 2019][2937616.847821] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937616.855600] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937616.863622] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:35 2019][2937616.870268] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:35 2019][2937616.877953] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:35 2019][2937616.884242] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:35 2019][2937616.890174] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:35 2019][2937616.896723] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:35 2019][2937616.902917] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:35 2019][2937616.909634] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:35 2019][2937616.916611] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:35 2019][2937616.922500] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937616.929599] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937616.937354] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:35 2019][2937616.944619] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937616.952494] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937616.959457] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:35 2019][2937616.965251] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937616.971732] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937616.979304] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:35 2019][2937616.984365] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937616.990641] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:35 2019][2937616.997262] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.003535] mdt02_014 S ffff8b6da9840000 0 118184 2 0x00000080 [Fri Apr 12 11:11:35 2019][2937617.010897] Call Trace: [Fri Apr 12 11:11:35 2019][2937617.013531] [] schedule+0x29/0x70 [Fri Apr 12 11:11:35 2019][2937617.018717] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.025595] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:11:35 2019][2937617.032563] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:35 2019][2937617.038357] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.044845] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.052418] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:35 2019][2937617.057478] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.063753] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:35 2019][2937617.070376] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.076647] mdt01_021 S ffff8b6da98430c0 0 118185 2 0x00000080 [Fri Apr 12 11:11:35 2019][2937617.084004] Call Trace: [Fri Apr 12 11:11:35 2019][2937617.086640] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:35 2019][2937617.093516] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:35 2019][2937617.099653] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.107401] [] schedule+0x29/0x70 [Fri Apr 12 11:11:35 2019][2937617.112583] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.119672] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.126756] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:35 2019][2937617.133896] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:35 2019][2937617.139686] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.147033] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.155076] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.162838] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:35 2019][2937617.169315] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.177096] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.185110] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:35 2019][2937617.191761] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:35 2019][2937617.199449] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:35 2019][2937617.205729] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:35 2019][2937617.211664] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:35 2019][2937617.218211] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:35 2019][2937617.224406] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:35 2019][2937617.231123] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:35 2019][2937617.238097] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:35 2019][2937617.243987] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.251076] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.258824] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:35 2019][2937617.266088] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.273954] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.280919] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:35 2019][2937617.287367] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:35 2019][2937617.293424] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.299911] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.307484] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:35 2019][2937617.312544] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.318819] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:35 2019][2937617.325439] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.331715] mdt00_020 S ffff8b7726bee180 0 118186 2 0x00000080 [Fri Apr 12 11:11:35 2019][2937617.339079] Call Trace: [Fri Apr 12 11:11:35 2019][2937617.341709] [] schedule+0x29/0x70 [Fri Apr 12 11:11:35 2019][2937617.346899] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.353783] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:35 2019][2937617.359576] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.366058] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.373629] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:35 2019][2937617.378712] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.384997] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:35 2019][2937617.391633] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.397930] mdt01_022 S ffff8b7726be8000 0 118187 2 0x00000080 [Fri Apr 12 11:11:35 2019][2937617.405312] Call Trace: [Fri Apr 12 11:11:35 2019][2937617.407954] [] schedule+0x29/0x70 [Fri Apr 12 11:11:35 2019][2937617.413156] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.420032] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:35 2019][2937617.425828] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.432355] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.439935] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:35 2019][2937617.445004] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.451307] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:35 2019][2937617.457958] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.464241] mdt02_015 S ffff8b7726be9040 0 118188 2 0x00000080 [Fri Apr 12 11:11:35 2019][2937617.471604] Call Trace: [Fri Apr 12 11:11:35 2019][2937617.474235] [] schedule+0x29/0x70 [Fri Apr 12 11:11:35 2019][2937617.479436] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.486326] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:35 2019][2937617.492123] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.498609] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.506181] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:35 2019][2937617.511240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.517546] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:35 2019][2937617.524165] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.530438] mdt01_023 S ffff8b7726bea080 0 118189 2 0x00000080 [Fri Apr 12 11:11:35 2019][2937617.537804] Call Trace: [Fri Apr 12 11:11:35 2019][2937617.540454] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:35 2019][2937617.547413] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.555185] [] schedule+0x29/0x70 [Fri Apr 12 11:11:35 2019][2937617.560401] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.567496] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.574592] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:35 2019][2937617.581734] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:35 2019][2937617.587533] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.594904] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.602982] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.610749] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:35 2019][2937617.617716] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.625518] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.633545] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:35 2019][2937617.640173] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:35 2019][2937617.646366] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:35 2019][2937617.653086] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:35 2019][2937617.660063] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:35 2019][2937617.665954] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.673050] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.680806] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:35 2019][2937617.688067] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.695936] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.702902] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:35 2019][2937617.709349] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:35 2019][2937617.715407] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.721890] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.729466] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:35 2019][2937617.734525] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.740800] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:35 2019][2937617.747422] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:35 2019][2937617.753695] mdt02_016 S ffff8b7726beb0c0 0 118191 2 0x00000080 [Fri Apr 12 11:11:35 2019][2937617.761058] Call Trace: [Fri Apr 12 11:11:35 2019][2937617.763724] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.771475] [] schedule+0x29/0x70 [Fri Apr 12 11:11:35 2019][2937617.776656] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.783743] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.790801] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:35 2019][2937617.796590] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.803844] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:35 2019][2937617.811192] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.818023] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:35 2019][2937617.826044] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:35 2019][2937617.832669] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:35 2019][2937617.839387] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:35 2019][2937617.845799] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:35 2019][2937617.852861] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:36 2019][2937617.860523] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:36 2019][2937617.866811] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:36 2019][2937617.874222] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:36 2019][2937617.881113] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:36 2019][2937617.887530] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937617.894696] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:36 2019][2937617.901680] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:36 2019][2937617.908332] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:36 2019][2937617.916020] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:36 2019][2937617.922307] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:36 2019][2937617.928242] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:36 2019][2937617.934797] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:36 2019][2937617.941000] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:36 2019][2937617.947721] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:36 2019][2937617.954703] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:36 2019][2937617.960606] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937617.967704] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937617.975466] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:36 2019][2937617.982748] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937617.990629] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937617.997603] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:36 2019][2937618.004093] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:36 2019][2937618.010152] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.016633] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.024207] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:36 2019][2937618.029268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.035545] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:36 2019][2937618.042175] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.048462] mdt02_017 S ffff8b7726bec100 0 118192 2 0x00000080 [Fri Apr 12 11:11:36 2019][2937618.055819] Call Trace: [Fri Apr 12 11:11:36 2019][2937618.058451] [] schedule+0x29/0x70 [Fri Apr 12 11:11:36 2019][2937618.063636] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.070516] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:36 2019][2937618.076306] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.082786] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.090365] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:36 2019][2937618.095440] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.101716] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:36 2019][2937618.108336] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.114610] mdt03_010 S ffff8b7cf7ffe180 0 118193 2 0x00000080 [Fri Apr 12 11:11:36 2019][2937618.121974] Call Trace: [Fri Apr 12 11:11:36 2019][2937618.124603] [] schedule+0x29/0x70 [Fri Apr 12 11:11:36 2019][2937618.129792] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.136669] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:36 2019][2937618.142484] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.148971] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.156557] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:36 2019][2937618.161624] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.167897] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:36 2019][2937618.174520] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.180792] mdt02_018 S ffff8b7cf7ff8000 0 118194 2 0x00000080 [Fri Apr 12 11:11:36 2019][2937618.188164] Call Trace: [Fri Apr 12 11:11:36 2019][2937618.190798] [] schedule+0x29/0x70 [Fri Apr 12 11:11:36 2019][2937618.195989] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.202867] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:36 2019][2937618.208659] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.215141] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.222712] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:36 2019][2937618.227787] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.234070] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:36 2019][2937618.240689] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.246963] mdt01_024 S ffff8b7cf7ffa080 0 118195 2 0x00000080 [Fri Apr 12 11:11:36 2019][2937618.254335] Call Trace: [Fri Apr 12 11:11:36 2019][2937618.256978] [] schedule+0x29/0x70 [Fri Apr 12 11:11:36 2019][2937618.262180] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.269089] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:36 2019][2937618.274896] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.281382] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.288955] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:36 2019][2937618.294017] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.300304] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:36 2019][2937618.306959] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.313229] mdt00_021 S ffff8b6da99a1040 0 118196 2 0x00000080 [Fri Apr 12 11:11:36 2019][2937618.320593] Call Trace: [Fri Apr 12 11:11:36 2019][2937618.323223] [] schedule+0x29/0x70 [Fri Apr 12 11:11:36 2019][2937618.328415] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.335296] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:36 2019][2937618.341092] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.347589] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.355161] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:36 2019][2937618.360223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.366513] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:36 2019][2937618.373146] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.379427] mdt03_011 S ffff8b6da99a30c0 0 118197 2 0x00000080 [Fri Apr 12 11:11:36 2019][2937618.386807] Call Trace: [Fri Apr 12 11:11:36 2019][2937618.388426] LNet: Service thread pid 117464 was inactive for 200.55s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 11:11:36 2019][2937618.388429] Pid: 117464, comm: mdt00_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:11:36 2019][2937618.388429] Call Trace: [Fri Apr 12 11:11:36 2019][2937618.388486] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.388519] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.388553] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.388561] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:36 2019][2937618.388571] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:36 2019][2937618.388581] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:36 2019][2937618.388593] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:36 2019][2937618.388603] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:36 2019][2937618.388614] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:36 2019][2937618.388627] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:36 2019][2937618.388640] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:36 2019][2937618.388651] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:36 2019][2937618.388661] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:36 2019][2937618.388711] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.388745] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.388778] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.388791] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:36 2019][2937618.388794] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:36 2019][2937618.388822] [] 0xffffffffffffffff [Fri Apr 12 11:11:36 2019][2937618.388825] LustreError: dumping log to /tmp/lustre-log.1555092696.117464 [Fri Apr 12 11:11:36 2019][2937618.550786] [] schedule+0x29/0x70 [Fri Apr 12 11:11:36 2019][2937618.555999] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.562885] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:36 2019][2937618.568706] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.575193] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.582772] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:36 2019][2937618.587833] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.594111] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:36 2019][2937618.600738] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:36 2019][2937618.607010] mdt02_019 S ffff8b7cf32430c0 0 118198 2 0x00000080 [Fri Apr 12 11:11:36 2019][2937618.614409] Call Trace: [Fri Apr 12 11:11:36 2019][2937618.617050] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:36 2019][2937618.623970] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.631776] [] schedule+0x29/0x70 [Fri Apr 12 11:11:36 2019][2937618.636968] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.644049] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.651136] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:36 2019][2937618.658295] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:36 2019][2937618.664083] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.671442] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.679476] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:36 2019][2937618.686204] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:36 2019][2937618.693179] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:36 2019][2937618.699938] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.707972] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:36 2019][2937618.715185] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.722345] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:11:36 2019][2937618.729524] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.736191] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.743284] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.750437] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:36 2019][2937618.757167] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:36 2019][2937618.763798] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:36 2019][2937618.770622] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.777514] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:36 2019][2937618.784835] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:36 2019][2937618.791406] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.798675] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.806398] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.812734] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.819844] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.827599] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:36 2019][2937618.834880] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.842757] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:36 2019][2937618.849729] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:36 2019][2937618.856174] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:37 2019][2937618.862229] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937618.868710] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937618.876294] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:37 2019][2937618.881355] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937618.887629] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:37 2019][2937618.894249] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937618.900540] mdt01_025 S ffff8b7cf3242080 0 118199 2 0x00000080 [Fri Apr 12 11:11:37 2019][2937618.907913] Call Trace: [Fri Apr 12 11:11:37 2019][2937618.910563] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:37 2019][2937618.917440] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:37 2019][2937618.923582] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937618.931325] [] schedule+0x29/0x70 [Fri Apr 12 11:11:37 2019][2937618.936519] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937618.943602] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937618.950679] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:37 2019][2937618.957823] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:37 2019][2937618.963627] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937618.970978] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937618.979026] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937618.986812] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:37 2019][2937618.993309] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.001114] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.009133] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.015803] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:37 2019][2937619.023490] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:37 2019][2937619.029778] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:37 2019][2937619.035746] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:37 2019][2937619.042304] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:37 2019][2937619.048505] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.055222] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.062195] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:37 2019][2937619.068105] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.075200] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.082957] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:37 2019][2937619.090217] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.098085] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.105054] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:37 2019][2937619.111499] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:37 2019][2937619.117553] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.124033] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.131607] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:37 2019][2937619.136669] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.142944] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:37 2019][2937619.149565] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.155839] mdt00_022 S ffff8b7cf3240000 0 118200 2 0x00000080 [Fri Apr 12 11:11:37 2019][2937619.163202] Call Trace: [Fri Apr 12 11:11:37 2019][2937619.165834] [] schedule+0x29/0x70 [Fri Apr 12 11:11:37 2019][2937619.171026] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.177937] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:37 2019][2937619.183753] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.190242] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.197821] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:37 2019][2937619.202905] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.209192] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:37 2019][2937619.215817] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.222101] mdt03_012 S ffff8b7cf3246180 0 118203 2 0x00000080 [Fri Apr 12 11:11:37 2019][2937619.229470] Call Trace: [Fri Apr 12 11:11:37 2019][2937619.232115] [] schedule+0x29/0x70 [Fri Apr 12 11:11:37 2019][2937619.237333] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.244230] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:37 2019][2937619.250032] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.256529] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.264110] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:37 2019][2937619.269197] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.275483] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:37 2019][2937619.282109] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.288393] mdt02_020 S ffff8b7cf3244100 0 118204 2 0x00000080 [Fri Apr 12 11:11:37 2019][2937619.295772] Call Trace: [Fri Apr 12 11:11:37 2019][2937619.298414] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:37 2019][2937619.305335] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.313083] [] schedule+0x29/0x70 [Fri Apr 12 11:11:37 2019][2937619.318268] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.325353] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.332443] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:37 2019][2937619.339606] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:37 2019][2937619.345409] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.352772] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.360789] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.367508] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:37 2019][2937619.374490] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.381236] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.389293] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.396388] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.403484] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.410650] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.417324] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:37 2019][2937619.424516] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.431184] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.438291] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.445452] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:37 2019][2937619.452181] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:37 2019][2937619.458825] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.465668] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.472556] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:37 2019][2937619.479878] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:37 2019][2937619.486455] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.493750] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.501474] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.507791] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.514889] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.522650] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:37 2019][2937619.529954] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.537855] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.544848] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:37 2019][2937619.551303] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:37 2019][2937619.557355] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.563847] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.571418] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:37 2019][2937619.576476] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.582754] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:37 2019][2937619.589386] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.595657] mdt01_026 S ffff8b7cf3245140 0 118205 2 0x00000080 [Fri Apr 12 11:11:37 2019][2937619.603065] Call Trace: [Fri Apr 12 11:11:37 2019][2937619.605715] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:37 2019][2937619.612632] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.620378] [] schedule+0x29/0x70 [Fri Apr 12 11:11:37 2019][2937619.625561] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.632655] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.639739] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:37 2019][2937619.646881] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:37 2019][2937619.652669] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.660018] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.668063] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.675827] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:37 2019][2937619.682764] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.690561] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.698582] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:37 2019][2937619.705212] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:37 2019][2937619.711407] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.718122] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:37 2019][2937619.725099] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:37 2019][2937619.730992] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.738087] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.745843] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:37 2019][2937619.753106] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.760980] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.767946] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:37 2019][2937619.774414] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:37 2019][2937619.780470] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.786952] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.794528] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:37 2019][2937619.799588] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.805865] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:37 2019][2937619.812513] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:37 2019][2937619.818801] mdt00_023 S ffff8b6da9a7b0c0 0 118206 2 0x00000080 [Fri Apr 12 11:11:37 2019][2937619.826166] Call Trace: [Fri Apr 12 11:11:37 2019][2937619.828806] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:37 2019][2937619.835724] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.843504] [] schedule+0x29/0x70 [Fri Apr 12 11:11:37 2019][2937619.848687] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.855772] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:37 2019][2937619.862856] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:38 2019][2937619.870000] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937619.875782] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.883125] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.891170] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.898948] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:38 2019][2937619.905892] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.913678] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.921702] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:38 2019][2937619.928340] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:38 2019][2937619.934552] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:38 2019][2937619.941270] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:38 2019][2937619.948244] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:38 2019][2937619.954140] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.961229] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.968980] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:38 2019][2937619.976249] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.984118] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937619.991084] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:38 2019][2937619.997533] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:38 2019][2937620.003596] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.010083] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.017657] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:38 2019][2937620.022725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.029002] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:38 2019][2937620.035622] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.041898] mdt01_027 S ffff8b6da9a7d140 0 118207 2 0x00000080 [Fri Apr 12 11:11:38 2019][2937620.049278] Call Trace: [Fri Apr 12 11:11:38 2019][2937620.051919] [] schedule+0x29/0x70 [Fri Apr 12 11:11:38 2019][2937620.057108] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.063988] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937620.069790] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.076268] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.083845] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:38 2019][2937620.088906] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.095183] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:38 2019][2937620.101802] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.108077] mdt00_024 S ffff8b6da9a7e180 0 118208 2 0x00000080 [Fri Apr 12 11:11:38 2019][2937620.115439] Call Trace: [Fri Apr 12 11:11:38 2019][2937620.118078] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:38 2019][2937620.124995] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.132744] [] schedule+0x29/0x70 [Fri Apr 12 11:11:38 2019][2937620.137923] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.145042] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.152124] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:38 2019][2937620.159266] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937620.165053] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.172410] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.180462] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.188228] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:38 2019][2937620.195143] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.202933] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.210958] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:38 2019][2937620.217607] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:38 2019][2937620.223807] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:38 2019][2937620.230526] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:38 2019][2937620.237519] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:38 2019][2937620.243415] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.250507] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.258274] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:38 2019][2937620.265558] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.273446] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.280427] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937620.286217] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.292698] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.300274] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:38 2019][2937620.305372] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.311646] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:38 2019][2937620.318267] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.324539] mdt02_021 S ffff8b6da9a7c100 0 118209 2 0x00000080 [Fri Apr 12 11:11:38 2019][2937620.331903] Call Trace: [Fri Apr 12 11:11:38 2019][2937620.334535] [] schedule+0x29/0x70 [Fri Apr 12 11:11:38 2019][2937620.339727] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.346608] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937620.352401] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.358882] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.366457] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:38 2019][2937620.371525] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.377799] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:38 2019][2937620.384420] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.390696] mdt00_025 S ffff8b6da9a78000 0 118210 2 0x00000080 [Fri Apr 12 11:11:38 2019][2937620.398065] Call Trace: [Fri Apr 12 11:11:38 2019][2937620.400696] [] schedule+0x29/0x70 [Fri Apr 12 11:11:38 2019][2937620.405899] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.412779] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937620.418575] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.425057] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.432631] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:38 2019][2937620.437685] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.443962] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:38 2019][2937620.450584] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.456858] mdt00_026 S ffff8b6da9a7a080 0 118211 2 0x00000080 [Fri Apr 12 11:11:38 2019][2937620.464219] Call Trace: [Fri Apr 12 11:11:38 2019][2937620.466852] [] schedule+0x29/0x70 [Fri Apr 12 11:11:38 2019][2937620.472040] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.478925] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937620.484716] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.491198] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.498770] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:38 2019][2937620.503830] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.510107] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:38 2019][2937620.516728] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.523002] mdt01_028 S ffff8b791671c100 0 118212 2 0x00000080 [Fri Apr 12 11:11:38 2019][2937620.530365] Call Trace: [Fri Apr 12 11:11:38 2019][2937620.533000] [] schedule+0x29/0x70 [Fri Apr 12 11:11:38 2019][2937620.538183] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.545079] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937620.550872] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.557359] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.564946] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:38 2019][2937620.570013] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.576286] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:38 2019][2937620.582917] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.589218] mdt00_027 S ffff8b7916719040 0 118213 2 0x00000080 [Fri Apr 12 11:11:38 2019][2937620.596582] Call Trace: [Fri Apr 12 11:11:38 2019][2937620.599220] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:38 2019][2937620.606102] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:38 2019][2937620.612241] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.619986] [] schedule+0x29/0x70 [Fri Apr 12 11:11:38 2019][2937620.625169] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.632263] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.639348] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:38 2019][2937620.646488] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:38 2019][2937620.652277] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.659609] [] ? mdt_version_save+0x67/0x120 [mdt] [Fri Apr 12 11:11:38 2019][2937620.666266] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.674304] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:38 2019][2937620.681040] [] mdt_reint_rename+0x13af/0x28e0 [mdt] [Fri Apr 12 11:11:38 2019][2937620.687762] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:38 2019][2937620.694503] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.702534] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:38 2019][2937620.708755] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:38 2019][2937620.715473] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:38 2019][2937620.722447] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:38 2019][2937620.728338] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.735448] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.743217] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:38 2019][2937620.750489] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.758368] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.765340] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:38 2019][2937620.771786] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:38 2019][2937620.777837] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.784316] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.791894] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:38 2019][2937620.796963] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.803244] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:38 2019][2937620.809869] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:38 2019][2937620.816159] mdt02_022 S ffff8b791671e180 0 118214 2 0x00000080 [Fri Apr 12 11:11:38 2019][2937620.823521] Call Trace: [Fri Apr 12 11:11:38 2019][2937620.826160] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:38 2019][2937620.833078] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.840827] [] schedule+0x29/0x70 [Fri Apr 12 11:11:38 2019][2937620.846010] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.853096] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:38 2019][2937620.860182] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:39 2019][2937620.867324] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:39 2019][2937620.873111] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.880459] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.888499] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.896269] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:39 2019][2937620.903183] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.910966] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.918990] [] mdt_reint_rename+0x7cc/0x28e0 [mdt] [Fri Apr 12 11:11:39 2019][2937620.925653] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.933673] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:39 2019][2937620.939874] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:39 2019][2937620.946589] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:39 2019][2937620.953575] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:39 2019][2937620.959473] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.966569] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.974324] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:39 2019][2937620.981597] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.989469] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937620.996440] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:39 2019][2937621.002234] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.008722] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.016300] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:39 2019][2937621.021367] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:39 2019][2937621.027643] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:39 2019][2937621.034263] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:39 2019][2937621.040537] mdt01_029 S ffff8b7916718000 0 118215 2 0x00000080 [Fri Apr 12 11:11:39 2019][2937621.047901] Call Trace: [Fri Apr 12 11:11:39 2019][2937621.050540] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.057461] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.065213] [] schedule+0x29/0x70 [Fri Apr 12 11:11:39 2019][2937621.070394] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.077486] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.084563] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:39 2019][2937621.091711] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:39 2019][2937621.097515] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.104862] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.112883] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.119600] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:39 2019][2937621.126573] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.133315] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.141375] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.148485] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.155566] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.162725] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.169366] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:39 2019][2937621.176544] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.183206] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.190294] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.197445] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:39 2019][2937621.204176] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:39 2019][2937621.210818] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.217664] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.224552] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.231875] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.238449] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.245712] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.253413] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.259730] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.266824] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.274574] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.281832] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.289708] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.296681] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:39 2019][2937621.302481] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.308973] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.316549] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:39 2019][2937621.321613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:39 2019][2937621.327888] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:39 2019][2937621.334518] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:39 2019][2937621.340790] mdt00_028 S ffff8b791671b0c0 0 118217 2 0x00000080 [Fri Apr 12 11:11:39 2019][2937621.348155] Call Trace: [Fri Apr 12 11:11:39 2019][2937621.350786] [] schedule+0x29/0x70 [Fri Apr 12 11:11:39 2019][2937621.355976] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.362867] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:39 2019][2937621.368687] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.375172] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.382768] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:39 2019][2937621.387829] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:39 2019][2937621.394100] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:39 2019][2937621.400722] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:39 2019][2937621.406995] mdt02_023 S ffff8b791671d140 0 118218 2 0x00000080 [Fri Apr 12 11:11:39 2019][2937621.414360] Call Trace: [Fri Apr 12 11:11:39 2019][2937621.416999] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.423919] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.431662] [] schedule+0x29/0x70 [Fri Apr 12 11:11:39 2019][2937621.436844] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.443923] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.450999] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:39 2019][2937621.458142] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:39 2019][2937621.463929] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.471281] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.479299] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.486014] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:39 2019][2937621.492993] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.499742] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.507787] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.514879] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.521939] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.529091] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.535720] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:39 2019][2937621.542899] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.549555] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.556642] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.563791] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:39 2019][2937621.570509] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:39 2019][2937621.577140] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.583970] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.590857] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.598176] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.604739] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.612004] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.619708] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.626022] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.633109] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.640861] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.648129] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.655993] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.662956] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:39 2019][2937621.669403] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:39 2019][2937621.675452] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.681938] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.689511] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:39 2019][2937621.694569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:39 2019][2937621.700840] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:39 2019][2937621.707459] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:39 2019][2937621.713732] mdt01_030 S ffff8b7ba4380000 0 118219 2 0x00000080 [Fri Apr 12 11:11:39 2019][2937621.721090] Call Trace: [Fri Apr 12 11:11:39 2019][2937621.723753] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.731501] [] schedule+0x29/0x70 [Fri Apr 12 11:11:39 2019][2937621.736682] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.743762] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.750820] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:39 2019][2937621.756608] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.763840] [] ? kfree+0x106/0x140 [Fri Apr 12 11:11:39 2019][2937621.769105] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.775923] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.783940] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:39 2019][2937621.790564] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:39 2019][2937621.797274] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:39 2019][2937621.803665] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:39 2019][2937621.810723] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:39 2019][2937621.818408] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:39 2019][2937621.825727] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:39 2019][2937621.832013] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:39 2019][2937621.839426] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:39 2019][2937621.846318] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.852718] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:39 2019][2937621.859993] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:39 2019][2937621.866979] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:40 2019][2937621.873655] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:40 2019][2937621.881341] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:40 2019][2937621.887627] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:40 2019][2937621.893566] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:40 2019][2937621.900111] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:40 2019][2937621.906307] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:40 2019][2937621.913023] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:40 2019][2937621.920000] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:40 2019][2937621.925893] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937621.932987] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937621.940744] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:40 2019][2937621.948015] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937621.955890] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937621.962857] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:40 2019][2937621.969305] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:40 2019][2937621.975355] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937621.981837] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937621.989410] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:40 2019][2937621.994470] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.000747] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:40 2019][2937622.007367] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.013641] mdt00_029 S ffff8b7ba4381040 0 118220 2 0x00000080 [Fri Apr 12 11:11:40 2019][2937622.021015] Call Trace: [Fri Apr 12 11:11:40 2019][2937622.023651] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:40 2019][2937622.030579] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.038338] [] schedule+0x29/0x70 [Fri Apr 12 11:11:40 2019][2937622.043529] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.050615] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.057696] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:40 2019][2937622.064843] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:40 2019][2937622.070637] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.078033] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.086109] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.093877] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:40 2019][2937622.100822] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.108634] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.116701] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:40 2019][2937622.123338] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:40 2019][2937622.129539] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:40 2019][2937622.136256] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:40 2019][2937622.143234] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:40 2019][2937622.149134] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.156231] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.163987] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:40 2019][2937622.171264] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.179190] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.186171] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:40 2019][2937622.192614] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:40 2019][2937622.198666] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.205147] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.212721] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:40 2019][2937622.217782] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.224078] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:40 2019][2937622.230698] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.236972] mdt01_031 S ffff8b7ba4382080 0 118221 2 0x00000080 [Fri Apr 12 11:11:40 2019][2937622.244336] Call Trace: [Fri Apr 12 11:11:40 2019][2937622.246984] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:40 2019][2937622.253905] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.261666] [] schedule+0x29/0x70 [Fri Apr 12 11:11:40 2019][2937622.266850] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.273953] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.281036] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:40 2019][2937622.288179] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:40 2019][2937622.293974] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.301322] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.309364] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.317153] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:40 2019][2937622.324078] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.331875] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.339899] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:40 2019][2937622.346544] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:40 2019][2937622.352765] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:40 2019][2937622.359505] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:40 2019][2937622.366487] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:40 2019][2937622.372412] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.379524] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.387280] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:40 2019][2937622.394547] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.402415] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.409383] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:40 2019][2937622.415831] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:40 2019][2937622.421882] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.428366] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.435947] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:40 2019][2937622.441009] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.447285] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:40 2019][2937622.453931] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.460201] mdt03_013 S ffff8b7ba4386180 0 118222 2 0x00000080 [Fri Apr 12 11:11:40 2019][2937622.467557] Call Trace: [Fri Apr 12 11:11:40 2019][2937622.470197] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:40 2019][2937622.477114] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.484866] [] schedule+0x29/0x70 [Fri Apr 12 11:11:40 2019][2937622.490055] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.497158] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.504245] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:40 2019][2937622.511385] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:40 2019][2937622.517175] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.524525] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.532599] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.540375] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:40 2019][2937622.547302] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.555090] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.563122] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:40 2019][2937622.569780] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:40 2019][2937622.575992] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:40 2019][2937622.582725] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:40 2019][2937622.589724] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:40 2019][2937622.595627] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.602744] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.610510] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:40 2019][2937622.617782] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.625696] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.632678] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:40 2019][2937622.639124] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:40 2019][2937622.645200] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.651696] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.659286] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:40 2019][2937622.664345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.670637] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:40 2019][2937622.677256] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.683530] mdt00_030 S ffff8b7ba43830c0 0 118223 2 0x00000080 [Fri Apr 12 11:11:40 2019][2937622.690893] Call Trace: [Fri Apr 12 11:11:40 2019][2937622.693524] [] schedule+0x29/0x70 [Fri Apr 12 11:11:40 2019][2937622.698717] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.705616] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:40 2019][2937622.711415] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.717940] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.725543] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:40 2019][2937622.730601] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.736889] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:40 2019][2937622.743525] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:40 2019][2937622.749826] mdt00_031 S ffff8b7ba4385140 0 118224 2 0x00000080 [Fri Apr 12 11:11:40 2019][2937622.757196] Call Trace: [Fri Apr 12 11:11:40 2019][2937622.759868] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:40 2019][2937622.766817] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.774588] [] schedule+0x29/0x70 [Fri Apr 12 11:11:40 2019][2937622.779769] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.786865] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.793951] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:40 2019][2937622.801101] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:40 2019][2937622.806887] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.814248] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.822294] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:40 2019][2937622.829046] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:40 2019][2937622.836029] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:40 2019][2937622.842793] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:40 2019][2937622.850851] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.857977] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.865072] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:41 2019][2937622.872243] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:41 2019][2937622.878894] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:41 2019][2937622.886079] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.892739] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.899871] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.907038] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:41 2019][2937622.913763] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:41 2019][2937622.920420] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:41 2019][2937622.927272] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.934189] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:41 2019][2937622.941517] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:41 2019][2937622.948088] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.955355] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.963056] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.969371] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.976462] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.984220] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:41 2019][2937622.991485] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937622.999357] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.006320] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:41 2019][2937623.012767] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:41 2019][2937623.018824] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.025311] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.032905] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:41 2019][2937623.037963] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.044237] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:41 2019][2937623.050857] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.057132] mdt01_032 S ffff8b8cee2a8000 0 118225 2 0x00000080 [Fri Apr 12 11:11:41 2019][2937623.064499] Call Trace: [Fri Apr 12 11:11:41 2019][2937623.067147] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:41 2019][2937623.074049] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:41 2019][2937623.080195] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.087949] [] schedule+0x29/0x70 [Fri Apr 12 11:11:41 2019][2937623.093140] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.100224] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.107311] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:41 2019][2937623.114455] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:41 2019][2937623.120250] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.127601] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.135655] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.143429] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.149922] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.157916] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.166001] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.172674] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:41 2019][2937623.180377] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:41 2019][2937623.186699] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:41 2019][2937623.192632] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:41 2019][2937623.199198] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:41 2019][2937623.205395] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.212132] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.219120] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:41 2019][2937623.225034] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.232144] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.239911] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:41 2019][2937623.247227] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.255100] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.262077] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:41 2019][2937623.267884] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.274380] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.281959] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:41 2019][2937623.287026] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.293301] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:41 2019][2937623.299919] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.306200] mdt00_032 S ffff8b8cee2a9040 0 118226 2 0x00000080 [Fri Apr 12 11:11:41 2019][2937623.313554] Call Trace: [Fri Apr 12 11:11:41 2019][2937623.316229] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.323991] [] schedule+0x29/0x70 [Fri Apr 12 11:11:41 2019][2937623.329172] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.336260] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.343315] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:41 2019][2937623.349102] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.356360] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:41 2019][2937623.363727] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.370581] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.378598] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:41 2019][2937623.385227] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:41 2019][2937623.391955] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:41 2019][2937623.398351] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:41 2019][2937623.405418] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:41 2019][2937623.413138] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:41 2019][2937623.420484] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:41 2019][2937623.426773] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:41 2019][2937623.434179] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:41 2019][2937623.441090] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.447495] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.454669] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.461654] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.468344] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:41 2019][2937623.476028] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:41 2019][2937623.482315] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:41 2019][2937623.488252] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:41 2019][2937623.494799] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:41 2019][2937623.501040] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.507788] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:41 2019][2937623.514788] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:41 2019][2937623.520696] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.527795] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.535570] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:41 2019][2937623.542847] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.550735] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.557729] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:41 2019][2937623.564192] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:41 2019][2937623.570246] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.576742] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.584344] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:41 2019][2937623.589442] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.595731] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:41 2019][2937623.602369] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.608657] mdt00_033 S ffff8b8cee2aa080 0 118227 2 0x00000080 [Fri Apr 12 11:11:41 2019][2937623.616030] Call Trace: [Fri Apr 12 11:11:41 2019][2937623.618710] [] schedule+0x29/0x70 [Fri Apr 12 11:11:41 2019][2937623.623906] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.630831] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:41 2019][2937623.636633] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.643187] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.650775] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:41 2019][2937623.655845] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.662135] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:41 2019][2937623.668758] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.675027] mdt03_014 S ffff8b8cee2ab0c0 0 118230 2 0x00000080 [Fri Apr 12 11:11:41 2019][2937623.682402] Call Trace: [Fri Apr 12 11:11:41 2019][2937623.685051] [] schedule+0x29/0x70 [Fri Apr 12 11:11:41 2019][2937623.690248] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.697134] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:41 2019][2937623.702923] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.709417] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.717004] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:41 2019][2937623.722076] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.728353] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:41 2019][2937623.735036] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.741305] mdt01_033 S ffff8b8cee2ac100 0 118232 2 0x00000080 [Fri Apr 12 11:11:41 2019][2937623.748676] Call Trace: [Fri Apr 12 11:11:41 2019][2937623.751307] [] schedule+0x29/0x70 [Fri Apr 12 11:11:41 2019][2937623.756500] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.763401] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:41 2019][2937623.769193] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.775681] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.783263] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:41 2019][2937623.788324] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.794627] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:41 2019][2937623.801247] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:41 2019][2937623.807525] mdt00_034 S ffff8b8cee2ad140 0 118233 2 0x00000080 [Fri Apr 12 11:11:41 2019][2937623.814899] Call Trace: [Fri Apr 12 11:11:41 2019][2937623.817541] [] schedule+0x29/0x70 [Fri Apr 12 11:11:41 2019][2937623.822753] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.829642] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:41 2019][2937623.835435] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.841924] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:41 2019][2937623.849541] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:41 2019][2937623.854609] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937623.860886] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937623.867504] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937623.873778] mdt03_015 S ffff8b8cee2ae180 0 118234 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937623.881138] Call Trace: [Fri Apr 12 11:11:42 2019][2937623.883772] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937623.888968] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937623.895882] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937623.901695] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937623.908189] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937623.915773] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:42 2019][2937623.920842] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937623.927134] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937623.933791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937623.940069] mdt00_035 S ffff8b85bcb68000 0 118235 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937623.947442] Call Trace: [Fri Apr 12 11:11:42 2019][2937623.950072] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937623.955266] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937623.962147] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937623.967939] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937623.974421] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937623.981991] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:42 2019][2937623.987053] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937623.993332] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937623.999951] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.006224] mdt03_016 S ffff8b85bcb69040 0 118236 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937624.013588] Call Trace: [Fri Apr 12 11:11:42 2019][2937624.016219] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937624.021409] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.028293] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937624.034087] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.040575] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.048146] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:42 2019][2937624.053206] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.059484] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937624.066103] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.072377] mdt01_034 S ffff8b85bcb6a080 0 118237 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937624.079731] Call Trace: [Fri Apr 12 11:11:42 2019][2937624.082370] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:42 2019][2937624.089255] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:42 2019][2937624.095392] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.103138] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937624.108320] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.115408] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.122494] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:42 2019][2937624.129631] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937624.135422] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.142769] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.150813] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.158574] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:42 2019][2937624.165065] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.172864] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.180886] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:42 2019][2937624.187541] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:42 2019][2937624.195240] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:42 2019][2937624.201565] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:42 2019][2937624.207539] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:42 2019][2937624.214102] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:42 2019][2937624.220299] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:42 2019][2937624.227015] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:42 2019][2937624.233990] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:42 2019][2937624.239886] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.246978] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.254728] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:42 2019][2937624.261993] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.269865] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.276853] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:42 2019][2937624.283334] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:42 2019][2937624.289395] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.295880] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.303454] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:42 2019][2937624.308517] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.314793] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937624.321410] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.327677] mdt02_024 S ffff8b85bcb6b0c0 0 118238 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937624.335034] Call Trace: [Fri Apr 12 11:11:42 2019][2937624.337671] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:42 2019][2937624.344590] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.352336] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937624.357517] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.364596] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.371672] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:42 2019][2937624.378814] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937624.384612] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.391969] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.400029] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.407799] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:42 2019][2937624.414729] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.422533] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.430558] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:42 2019][2937624.437188] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:42 2019][2937624.443388] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:42 2019][2937624.450109] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:42 2019][2937624.457105] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:42 2019][2937624.462995] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.470090] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.477838] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:42 2019][2937624.485099] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.492966] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.499933] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:42 2019][2937624.506379] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:42 2019][2937624.512445] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.518933] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.526518] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:42 2019][2937624.531576] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.537862] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937624.544482] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.550754] mdt03_017 S ffff8b85bcb6c100 0 118239 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937624.558171] Call Trace: [Fri Apr 12 11:11:42 2019][2937624.560801] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937624.565997] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.572875] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937624.578668] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.585157] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.592730] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:42 2019][2937624.597790] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.604076] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937624.610698] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.616968] mdt01_035 S ffff8b85bcb6d140 0 118240 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937624.624384] Call Trace: [Fri Apr 12 11:11:42 2019][2937624.627016] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937624.632206] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.640250] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.647217] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:42 2019][2937624.653667] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937624.659483] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.666164] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.673802] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:11:42 2019][2937624.679035] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.685311] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937624.692104] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.698375] mdt00_036 S ffff8b85bcb6e180 0 118241 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937624.705731] Call Trace: [Fri Apr 12 11:11:42 2019][2937624.708363] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937624.713561] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.720443] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937624.726238] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.732719] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.740290] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:42 2019][2937624.745351] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.751627] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:42 2019][2937624.758248] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:42 2019][2937624.764521] mdt01_036 S ffff8b85322e8000 0 118242 2 0x00000080 [Fri Apr 12 11:11:42 2019][2937624.771878] Call Trace: [Fri Apr 12 11:11:42 2019][2937624.774514] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:42 2019][2937624.781436] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.789182] [] schedule+0x29/0x70 [Fri Apr 12 11:11:42 2019][2937624.794365] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.801452] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.808535] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:42 2019][2937624.815683] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:42 2019][2937624.821471] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.828815] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:42 2019][2937624.836852] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:42 2019][2937624.843612] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:42 2019][2937624.850589] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:42 2019][2937624.857336] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.865388] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.872487] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.879556] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:43 2019][2937624.886704] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:43 2019][2937624.893332] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:43 2019][2937624.900512] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.907167] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.914258] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.921405] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:43 2019][2937624.928121] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:43 2019][2937624.934774] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.941666] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:43 2019][2937624.948491] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.955377] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:43 2019][2937624.962700] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:43 2019][2937624.969270] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.976537] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.984241] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.990558] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937624.997657] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.005410] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:43 2019][2937625.012677] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.020546] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.027510] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:43 2019][2937625.033959] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:43 2019][2937625.040013] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.046494] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.054068] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:43 2019][2937625.059127] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:43 2019][2937625.065402] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:43 2019][2937625.072023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:43 2019][2937625.078316] mdt00_037 D ffff8b85322e9040 0 118243 2 0x00000080 [Fri Apr 12 11:11:43 2019][2937625.085679] Call Trace: [Fri Apr 12 11:11:43 2019][2937625.088312] [] schedule_preempt_disabled+0x29/0x70 [Fri Apr 12 11:11:43 2019][2937625.094930] [] __mutex_lock_slowpath+0xc7/0x1d0 [Fri Apr 12 11:11:43 2019][2937625.101291] [] mutex_lock+0x1f/0x2f [Fri Apr 12 11:11:43 2019][2937625.106661] [] tgt_lookup_reply_by_xid+0x19/0x60 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.113926] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.120809] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:11:43 2019][2937625.126563] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:11:43 2019][2937625.133190] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:11:43 2019][2937625.139376] [] ? schedule+0x29/0x70 [Fri Apr 12 11:11:43 2019][2937625.144745] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.151823] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:43 2019][2937625.158267] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:43 2019][2937625.164322] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.170989] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.178585] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:11:43 2019][2937625.183818] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:43 2019][2937625.190105] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:43 2019][2937625.196933] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:43 2019][2937625.203207] mdt02_025 S ffff8b85322ea080 0 118245 2 0x00000080 [Fri Apr 12 11:11:43 2019][2937625.210578] Call Trace: [Fri Apr 12 11:11:43 2019][2937625.213258] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.221027] [] schedule+0x29/0x70 [Fri Apr 12 11:11:43 2019][2937625.226218] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.233314] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.240384] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:43 2019][2937625.246204] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.253468] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:43 2019][2937625.260814] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.267641] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.275674] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:43 2019][2937625.282302] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:43 2019][2937625.289035] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:43 2019][2937625.295453] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:43 2019][2937625.302519] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:43 2019][2937625.310214] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:43 2019][2937625.317541] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:43 2019][2937625.323853] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:43 2019][2937625.331296] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:43 2019][2937625.338187] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.344607] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.351771] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.358770] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.365432] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:43 2019][2937625.373118] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:43 2019][2937625.379406] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:43 2019][2937625.385340] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:43 2019][2937625.391889] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:43 2019][2937625.398101] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.404834] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.411824] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:43 2019][2937625.417727] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.424835] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.432593] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:43 2019][2937625.439881] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.447757] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.454730] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:43 2019][2937625.461185] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:43 2019][2937625.467250] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.473767] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.481372] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:43 2019][2937625.486432] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:43 2019][2937625.492705] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:43 2019][2937625.499326] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:43 2019][2937625.505598] mdt03_018 S ffff8b85322eb0c0 0 118246 2 0x00000080 [Fri Apr 12 11:11:43 2019][2937625.512974] Call Trace: [Fri Apr 12 11:11:43 2019][2937625.515613] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:43 2019][2937625.522532] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.530296] [] schedule+0x29/0x70 [Fri Apr 12 11:11:43 2019][2937625.535529] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.542626] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.549710] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:43 2019][2937625.556859] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:43 2019][2937625.562647] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.570015] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.578044] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.584759] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:43 2019][2937625.591761] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.598515] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.606609] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.613712] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.620789] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.627957] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.634585] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:43 2019][2937625.641769] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.648447] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.655540] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.662710] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:43 2019][2937625.669443] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:43 2019][2937625.676090] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:43 2019][2937625.682920] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.689839] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:43 2019][2937625.697175] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:43 2019][2937625.703768] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.711046] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.718768] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.725111] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.732212] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.739977] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:43 2019][2937625.747257] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.755152] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.762136] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:43 2019][2937625.768591] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:43 2019][2937625.774703] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.781186] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.788770] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:43 2019][2937625.793826] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:43 2019][2937625.800101] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:43 2019][2937625.806723] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:43 2019][2937625.812994] mdt00_038 S ffff8b85322ec100 0 118247 2 0x00000080 [Fri Apr 12 11:11:43 2019][2937625.820352] Call Trace: [Fri Apr 12 11:11:43 2019][2937625.822982] [] schedule+0x29/0x70 [Fri Apr 12 11:11:43 2019][2937625.828176] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.835055] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:43 2019][2937625.840865] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.847348] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:43 2019][2937625.854918] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:43 2019][2937625.859978] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937625.866247] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:44 2019][2937625.872868] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937625.879140] mdt01_037 S ffff8b85322ed140 0 118248 2 0x00000080 [Fri Apr 12 11:11:44 2019][2937625.886497] Call Trace: [Fri Apr 12 11:11:44 2019][2937625.889132] [] schedule+0x29/0x70 [Fri Apr 12 11:11:44 2019][2937625.894328] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937625.901262] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:44 2019][2937625.907058] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937625.913555] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937625.921152] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:44 2019][2937625.926214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937625.932501] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:44 2019][2937625.939117] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937625.945392] mdt02_026 D ffff8b85322ee180 0 118249 2 0x00000080 [Fri Apr 12 11:11:44 2019][2937625.952755] Call Trace: [Fri Apr 12 11:11:44 2019][2937625.955386] [] ? physflat_send_IPI_mask+0xe/0x10 [Fri Apr 12 11:11:44 2019][2937625.961833] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:11:44 2019][2937625.967587] [] ? account_entity_dequeue+0xae/0xd0 [Fri Apr 12 11:11:44 2019][2937625.974122] [] ? dequeue_entity+0x11c/0x5e0 [Fri Apr 12 11:11:44 2019][2937625.980136] [] ? __enqueue_entity+0x78/0x80 [Fri Apr 12 11:11:44 2019][2937625.986151] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:11:44 2019][2937625.992428] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:11:44 2019][2937625.998356] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:11:44 2019][2937626.004111] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:11:44 2019][2937626.010732] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:11:44 2019][2937626.016920] [] ? schedule+0x29/0x70 [Fri Apr 12 11:11:44 2019][2937626.022298] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.029347] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:44 2019][2937626.035142] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.041795] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.049369] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:11:44 2019][2937626.054604] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937626.060917] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:44 2019][2937626.067719] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937626.074009] mdt01_038 S ffff8b8cd1cf8000 0 118250 2 0x00000080 [Fri Apr 12 11:11:44 2019][2937626.081380] Call Trace: [Fri Apr 12 11:11:44 2019][2937626.084022] [] schedule+0x29/0x70 [Fri Apr 12 11:11:44 2019][2937626.089167] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:11:44 2019][2937626.095232] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:44 2019][2937626.102552] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:11:44 2019][2937626.109105] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Fri Apr 12 11:11:44 2019][2937626.116026] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.123821] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.130937] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.138008] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:44 2019][2937626.143805] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.151076] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:44 2019][2937626.158428] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.165263] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.173282] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:44 2019][2937626.179911] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:44 2019][2937626.186639] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:44 2019][2937626.193028] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:44 2019][2937626.200096] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:44 2019][2937626.207779] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:44 2019][2937626.215105] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:44 2019][2937626.221392] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:44 2019][2937626.228800] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:44 2019][2937626.235700] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.242106] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.249255] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.256232] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.262881] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:44 2019][2937626.270586] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:44 2019][2937626.276869] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:44 2019][2937626.282806] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:44 2019][2937626.289366] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:44 2019][2937626.295589] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.302315] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.309291] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:44 2019][2937626.315239] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.322365] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.330140] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:44 2019][2937626.337408] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.345274] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.352244] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:44 2019][2937626.358689] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:44 2019][2937626.364754] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.371239] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.378842] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:44 2019][2937626.383903] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937626.390179] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:44 2019][2937626.396802] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937626.403071] mdt00_039 S ffff8b8cd1cf9040 0 118251 2 0x00000080 [Fri Apr 12 11:11:44 2019][2937626.410439] Call Trace: [Fri Apr 12 11:11:44 2019][2937626.413087] [] schedule+0x29/0x70 [Fri Apr 12 11:11:44 2019][2937626.418300] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.425187] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:44 2019][2937626.430980] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.437469] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.445057] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:44 2019][2937626.450118] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937626.456402] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:44 2019][2937626.463023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937626.469332] mdt02_027 S ffff8b8cd1cfa080 0 118252 2 0x00000080 [Fri Apr 12 11:11:44 2019][2937626.476693] Call Trace: [Fri Apr 12 11:11:44 2019][2937626.479344] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:44 2019][2937626.486291] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.494045] [] schedule+0x29/0x70 [Fri Apr 12 11:11:44 2019][2937626.499241] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.506328] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.513451] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:44 2019][2937626.520599] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:44 2019][2937626.526453] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.533807] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.541875] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.549652] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:44 2019][2937626.556597] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.564378] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.572403] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:44 2019][2937626.579051] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:44 2019][2937626.580589] LNet: Service thread pid 118263 was inactive for 200.62s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 12 11:11:44 2019][2937626.580592] Pid: 118263, comm: mdt00_044 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:11:44 2019][2937626.580592] Call Trace: [Fri Apr 12 11:11:44 2019][2937626.580654] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.580687] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.580716] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.580724] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:44 2019][2937626.580734] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:44 2019][2937626.580743] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:44 2019][2937626.580764] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:44 2019][2937626.580774] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:44 2019][2937626.580786] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.580796] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.580809] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:44 2019][2937626.580819] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.580828] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:44 2019][2937626.580875] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.580909] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.580941] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.580945] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:44 2019][2937626.580949] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:44 2019][2937626.580976] [] 0xffffffffffffffff [Fri Apr 12 11:11:44 2019][2937626.580980] LustreError: dumping log to /tmp/lustre-log.1555092704.118263 [Fri Apr 12 11:11:44 2019][2937626.746634] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.753359] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:44 2019][2937626.760367] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:44 2019][2937626.766268] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.773371] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.781131] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:44 2019][2937626.788399] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.796307] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.803276] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:44 2019][2937626.809742] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:44 2019][2937626.815797] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.822283] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:44 2019][2937626.829899] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:44 2019][2937626.834961] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937626.841238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:44 2019][2937626.847866] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:44 2019][2937626.854138] mdt00_040 S ffff8b8cd1cfb0c0 0 118253 2 0x00000080 [Fri Apr 12 11:11:44 2019][2937626.861500] Call Trace: [Fri Apr 12 11:11:44 2019][2937626.864152] [] schedule+0x29/0x70 [Fri Apr 12 11:11:45 2019][2937626.869351] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937626.876233] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:45 2019][2937626.882144] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937626.888668] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937626.896248] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:45 2019][2937626.901316] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:45 2019][2937626.907593] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:45 2019][2937626.914210] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:45 2019][2937626.920510] mdt01_039 S ffff8b8cd1cfc100 0 118254 2 0x00000080 [Fri Apr 12 11:11:45 2019][2937626.927872] Call Trace: [Fri Apr 12 11:11:45 2019][2937626.930504] [] schedule+0x29/0x70 [Fri Apr 12 11:11:45 2019][2937626.935692] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937626.942569] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:45 2019][2937626.948361] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937626.954843] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937626.962417] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:45 2019][2937626.967477] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:45 2019][2937626.973753] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:45 2019][2937626.980382] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:45 2019][2937626.986656] mdt03_019 S ffff8b8cd1cfd140 0 118255 2 0x00000080 [Fri Apr 12 11:11:45 2019][2937626.994045] Call Trace: [Fri Apr 12 11:11:45 2019][2937626.996730] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.004504] [] schedule+0x29/0x70 [Fri Apr 12 11:11:45 2019][2937627.009694] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.016821] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.023885] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:45 2019][2937627.029722] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.036979] [] ? kfree+0x106/0x140 [Fri Apr 12 11:11:45 2019][2937627.042248] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.049074] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.057104] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:45 2019][2937627.063730] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:45 2019][2937627.070458] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:45 2019][2937627.076894] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:45 2019][2937627.083961] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:45 2019][2937627.091647] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:45 2019][2937627.098982] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:45 2019][2937627.105267] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:45 2019][2937627.112701] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:45 2019][2937627.119619] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.126023] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.133180] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.140177] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.146838] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:45 2019][2937627.154553] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:45 2019][2937627.160837] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:45 2019][2937627.166792] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:45 2019][2937627.173335] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:45 2019][2937627.179529] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.186260] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.193250] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:45 2019][2937627.199158] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.206278] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.214036] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:45 2019][2937627.221302] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.229177] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.236143] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:45 2019][2937627.242608] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:45 2019][2937627.248664] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.255150] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.262725] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:45 2019][2937627.267784] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:45 2019][2937627.274060] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:45 2019][2937627.280679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:45 2019][2937627.286952] mdt01_040 S ffff8b8cd1cfe180 0 118256 2 0x00000080 [Fri Apr 12 11:11:45 2019][2937627.294361] Call Trace: [Fri Apr 12 11:11:45 2019][2937627.296999] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:45 2019][2937627.303938] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.311703] [] schedule+0x29/0x70 [Fri Apr 12 11:11:45 2019][2937627.316882] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.323967] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.331052] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:45 2019][2937627.338195] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:45 2019][2937627.343982] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.351331] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.359385] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.366104] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:45 2019][2937627.373081] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.379843] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.387903] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.395113] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.402266] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:11:45 2019][2937627.409468] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.416129] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.423215] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.430368] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:45 2019][2937627.437084] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:45 2019][2937627.443723] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.450550] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.457451] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:45 2019][2937627.464777] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:45 2019][2937627.471346] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.478614] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.486322] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.492658] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.499760] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.507536] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:45 2019][2937627.514814] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.522721] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.529710] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:45 2019][2937627.536182] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:45 2019][2937627.542238] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.548733] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.556314] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:45 2019][2937627.561373] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:45 2019][2937627.567647] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:45 2019][2937627.574267] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:45 2019][2937627.580542] mdt00_041 S ffff8b89deb08000 0 118257 2 0x00000080 [Fri Apr 12 11:11:45 2019][2937627.587898] Call Trace: [Fri Apr 12 11:11:45 2019][2937627.590536] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:45 2019][2937627.597454] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.604612] Pid: 118237, comm: mdt01_034 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 12 11:11:45 2019][2937627.604613] Call Trace: [Fri Apr 12 11:11:45 2019][2937627.604674] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.604703] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.604717] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.604728] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.604740] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:45 2019][2937627.604749] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.604759] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:45 2019][2937627.604806] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.604846] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.604879] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.604883] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:45 2019][2937627.604887] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:45 2019][2937627.604915] [] 0xffffffffffffffff [Fri Apr 12 11:11:45 2019][2937627.604917] LustreError: dumping log to /tmp/lustre-log.1555092705.118237 [Fri Apr 12 11:11:45 2019][2937627.709070] [] schedule+0x29/0x70 [Fri Apr 12 11:11:45 2019][2937627.714278] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.721440] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.728557] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:45 2019][2937627.735702] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:45 2019][2937627.741488] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.748838] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.756856] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.763574] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:45 2019][2937627.770549] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.777308] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.785356] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.792421] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.799605] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.807025] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:11:45 2019][2937627.814122] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.820847] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.827948] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.835129] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:45 2019][2937627.841854] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:45 2019][2937627.848509] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:45 2019][2937627.855345] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:45 2019][2937627.862236] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:45 2019][2937627.869560] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:46 2019][2937627.876132] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.883421] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.891178] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.897535] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.904635] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.912385] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:46 2019][2937627.919651] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.927586] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.934554] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:46 2019][2937627.941002] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:46 2019][2937627.947063] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.953588] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937627.961161] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:46 2019][2937627.966221] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937627.972515] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:46 2019][2937627.979135] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937627.985417] mdt02_028 S ffff8b89deb09040 0 118258 2 0x00000080 [Fri Apr 12 11:11:46 2019][2937627.992806] Call Trace: [Fri Apr 12 11:11:46 2019][2937627.995458] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.002383] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.010140] [] schedule+0x29/0x70 [Fri Apr 12 11:11:46 2019][2937628.015343] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.022438] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.029526] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:46 2019][2937628.036685] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:46 2019][2937628.042471] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.049819] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.057861] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.065634] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:46 2019][2937628.072552] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.080332] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.088368] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:46 2019][2937628.095020] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:46 2019][2937628.101224] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.107944] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.114917] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:46 2019][2937628.120850] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.127941] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.135703] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.142964] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.150844] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.157822] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:46 2019][2937628.164328] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:46 2019][2937628.170405] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.176887] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.184474] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:46 2019][2937628.189533] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937628.195805] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:46 2019][2937628.202457] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937628.208728] mdt00_042 S ffff8b89deb0a080 0 118259 2 0x00000080 [Fri Apr 12 11:11:46 2019][2937628.216099] Call Trace: [Fri Apr 12 11:11:46 2019][2937628.218737] [] schedule+0x29/0x70 [Fri Apr 12 11:11:46 2019][2937628.223936] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.230871] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:46 2019][2937628.236726] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.243243] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.250832] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:46 2019][2937628.255894] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937628.262180] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:46 2019][2937628.268798] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937628.275079] mdt00_043 S ffff8b89deb0b0c0 0 118260 2 0x00000080 [Fri Apr 12 11:11:46 2019][2937628.282443] Call Trace: [Fri Apr 12 11:11:46 2019][2937628.285085] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.292027] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.299773] [] schedule+0x29/0x70 [Fri Apr 12 11:11:46 2019][2937628.304954] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.312069] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.319168] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:46 2019][2937628.326323] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:46 2019][2937628.332108] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.339457] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.347500] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.355267] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:46 2019][2937628.362182] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.369963] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.377986] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:46 2019][2937628.384619] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:46 2019][2937628.390827] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.397545] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.404521] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:46 2019][2937628.410412] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.417510] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.425284] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.432542] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.440413] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.447378] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:46 2019][2937628.453825] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:46 2019][2937628.459880] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.466359] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.473934] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:46 2019][2937628.478994] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937628.485271] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:46 2019][2937628.491889] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937628.498165] mdt01_041 S ffff8b89deb0c100 0 118261 2 0x00000080 [Fri Apr 12 11:11:46 2019][2937628.505526] Call Trace: [Fri Apr 12 11:11:46 2019][2937628.508165] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.515086] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.522833] [] schedule+0x29/0x70 [Fri Apr 12 11:11:46 2019][2937628.528012] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.535100] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.542174] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:46 2019][2937628.549317] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:46 2019][2937628.555105] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.562453] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.570474] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.577192] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:46 2019][2937628.584167] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.590909] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.598940] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.606126] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.613292] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:11:46 2019][2937628.620470] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.627131] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.634223] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.641394] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:46 2019][2937628.648141] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:46 2019][2937628.654782] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.661646] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.668541] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.675873] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.682444] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.689707] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.697457] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.703782] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.710891] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.718643] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.725911] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.733798] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.740774] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:46 2019][2937628.746576] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.753081] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.760663] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:46 2019][2937628.765722] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937628.772008] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:46 2019][2937628.778630] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:46 2019][2937628.784901] mdt02_029 S ffff8b89deb0d140 0 118262 2 0x00000080 [Fri Apr 12 11:11:46 2019][2937628.792265] Call Trace: [Fri Apr 12 11:11:46 2019][2937628.794912] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:46 2019][2937628.801833] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.809594] [] schedule+0x29/0x70 [Fri Apr 12 11:11:46 2019][2937628.814777] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.821870] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.828976] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:46 2019][2937628.836119] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:46 2019][2937628.841936] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.849287] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:46 2019][2937628.857359] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:46 2019][2937628.864074] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:46 2019][2937628.871069] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:47 2019][2937628.877820] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.885875] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.892974] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.900042] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:47 2019][2937628.907202] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:47 2019][2937628.913831] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:47 2019][2937628.921046] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.927711] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.934803] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.941954] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:47 2019][2937628.948671] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:47 2019][2937628.955300] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:47 2019][2937628.962129] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.969018] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:47 2019][2937628.976384] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:47 2019][2937628.982974] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.990242] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937628.997966] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.004285] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.011400] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.019175] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:47 2019][2937629.026441] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.034411] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.041382] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:47 2019][2937629.047831] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:47 2019][2937629.053917] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.060425] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.067995] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:47 2019][2937629.073077] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:47 2019][2937629.079362] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:47 2019][2937629.085997] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:47 2019][2937629.092268] mdt00_044 S ffff8b89deb0e180 0 118263 2 0x00000080 [Fri Apr 12 11:11:47 2019][2937629.099633] Call Trace: [Fri Apr 12 11:11:47 2019][2937629.102302] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.110088] [] schedule+0x29/0x70 [Fri Apr 12 11:11:47 2019][2937629.115273] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.122388] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.129466] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:47 2019][2937629.135269] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.142525] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.149873] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.156707] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.164775] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:47 2019][2937629.171461] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:47 2019][2937629.178235] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:47 2019][2937629.184637] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.191697] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:47 2019][2937629.199382] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.206709] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:47 2019][2937629.212999] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:47 2019][2937629.220415] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:47 2019][2937629.227308] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.233716] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.240866] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.247844] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.254494] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.262174] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.268463] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:47 2019][2937629.274400] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:47 2019][2937629.280946] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:47 2019][2937629.287159] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.293930] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.300911] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:47 2019][2937629.306806] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.313898] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.321654] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:47 2019][2937629.328917] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.336791] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.343758] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:47 2019][2937629.350203] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:47 2019][2937629.356260] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.362739] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.370313] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:47 2019][2937629.375373] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:47 2019][2937629.381649] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:47 2019][2937629.388269] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:47 2019][2937629.394542] mdt01_042 S ffff8b867e3d8000 0 118264 2 0x00000080 [Fri Apr 12 11:11:47 2019][2937629.401899] Call Trace: [Fri Apr 12 11:11:47 2019][2937629.404563] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.412311] [] schedule+0x29/0x70 [Fri Apr 12 11:11:47 2019][2937629.417491] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.424574] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.431632] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:47 2019][2937629.437419] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.444673] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.452025] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.458849] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.466862] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:47 2019][2937629.473488] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:47 2019][2937629.480199] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:47 2019][2937629.486588] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.493649] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:47 2019][2937629.501334] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.508650] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:47 2019][2937629.514931] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:47 2019][2937629.522338] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:47 2019][2937629.529231] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.535636] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.542785] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.549762] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.556412] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.564092] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.570380] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:47 2019][2937629.576315] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:47 2019][2937629.582861] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:47 2019][2937629.589056] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.595774] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.602751] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:47 2019][2937629.608643] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.615738] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.623484] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:47 2019][2937629.630746] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.638616] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.645582] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:47 2019][2937629.652059] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:47 2019][2937629.658126] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.664610] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.672190] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:47 2019][2937629.677250] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:47 2019][2937629.683526] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:47 2019][2937629.690146] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:47 2019][2937629.696439] mdt01_043 S ffff8b867e3d9040 0 118265 2 0x00000080 [Fri Apr 12 11:11:47 2019][2937629.703879] Call Trace: [Fri Apr 12 11:11:47 2019][2937629.706537] [] schedule+0x29/0x70 [Fri Apr 12 11:11:47 2019][2937629.711749] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.718653] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:47 2019][2937629.724446] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.730935] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.738519] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:47 2019][2937629.743578] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:47 2019][2937629.749866] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:47 2019][2937629.756494] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:47 2019][2937629.762767] mdt02_030 S ffff8b867e3da080 0 118266 2 0x00000080 [Fri Apr 12 11:11:47 2019][2937629.770206] Call Trace: [Fri Apr 12 11:11:47 2019][2937629.772846] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:47 2019][2937629.779789] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.787537] [] schedule+0x29/0x70 [Fri Apr 12 11:11:47 2019][2937629.792718] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.799801] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.806881] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:47 2019][2937629.814021] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:47 2019][2937629.819810] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.827160] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:47 2019][2937629.835182] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:47 2019][2937629.841933] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:47 2019][2937629.848906] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:48 2019][2937629.855651] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.863695] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.870805] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.877885] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:48 2019][2937629.885039] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:48 2019][2937629.891671] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:48 2019][2937629.898893] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.905582] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.912698] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.919854] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:48 2019][2937629.926582] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:48 2019][2937629.933236] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.940125] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:48 2019][2937629.946952] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.953855] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:48 2019][2937629.961190] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:48 2019][2937629.967782] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.975091] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.982793] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.989119] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937629.996210] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.003966] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:48 2019][2937630.011240] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.019117] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.026087] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:48 2019][2937630.032529] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:48 2019][2937630.038585] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.045078] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.052677] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:48 2019][2937630.057744] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.064018] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:48 2019][2937630.070641] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.076943] mdt00_045 S ffff8b867e3db0c0 0 118267 2 0x00000080 [Fri Apr 12 11:11:48 2019][2937630.084318] Call Trace: [Fri Apr 12 11:11:48 2019][2937630.086960] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:48 2019][2937630.093851] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:48 2019][2937630.099999] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.107763] [] schedule+0x29/0x70 [Fri Apr 12 11:11:48 2019][2937630.112942] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.120063] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.127149] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:48 2019][2937630.134289] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:48 2019][2937630.140081] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.147444] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.155484] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.163239] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.169717] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.177513] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.185530] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.192175] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:48 2019][2937630.199879] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:48 2019][2937630.206178] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:48 2019][2937630.212117] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:48 2019][2937630.218672] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:48 2019][2937630.224877] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.231604] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.238577] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:48 2019][2937630.244474] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.251572] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.259331] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:48 2019][2937630.266608] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.274506] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.281480] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:48 2019][2937630.287923] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:48 2019][2937630.293981] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.300480] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.308061] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:48 2019][2937630.313154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.319457] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:48 2019][2937630.326105] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.332388] mdt03_020 S ffff8b867e3dc100 0 118268 2 0x00000080 [Fri Apr 12 11:11:48 2019][2937630.339757] Call Trace: [Fri Apr 12 11:11:48 2019][2937630.342389] [] schedule+0x29/0x70 [Fri Apr 12 11:11:48 2019][2937630.347591] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.354474] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:48 2019][2937630.360272] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.366756] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.374335] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:48 2019][2937630.379394] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.385671] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:48 2019][2937630.392312] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.398656] mdt01_044 S ffff8b867e3dd140 0 118269 2 0x00000080 [Fri Apr 12 11:11:48 2019][2937630.406076] Call Trace: [Fri Apr 12 11:11:48 2019][2937630.408713] [] schedule+0x29/0x70 [Fri Apr 12 11:11:48 2019][2937630.413912] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.420799] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:48 2019][2937630.426614] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.433099] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.440681] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:48 2019][2937630.445740] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.452016] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:48 2019][2937630.458636] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.464908] mdt00_046 S ffff8b867e3de180 0 118270 2 0x00000080 [Fri Apr 12 11:11:48 2019][2937630.472265] Call Trace: [Fri Apr 12 11:11:48 2019][2937630.474896] [] schedule+0x29/0x70 [Fri Apr 12 11:11:48 2019][2937630.480088] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.486968] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:11:48 2019][2937630.493935] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:48 2019][2937630.499731] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.506211] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.513783] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:48 2019][2937630.518844] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.525119] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:48 2019][2937630.531751] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:48 2019][2937630.538039] mdt01_045 S ffff8b8c24e40000 0 118271 2 0x00000080 [Fri Apr 12 11:11:48 2019][2937630.545395] Call Trace: [Fri Apr 12 11:11:48 2019][2937630.548089] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.555841] [] schedule+0x29/0x70 [Fri Apr 12 11:11:48 2019][2937630.561027] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.568151] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.575212] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:48 2019][2937630.581033] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.588288] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:48 2019][2937630.595638] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.602465] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.610495] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:48 2019][2937630.617128] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:48 2019][2937630.623845] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:48 2019][2937630.630236] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:48 2019][2937630.637293] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:48 2019][2937630.644982] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:48 2019][2937630.652304] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:48 2019][2937630.658596] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:48 2019][2937630.666083] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:48 2019][2937630.673083] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.679513] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.686669] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.693674] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.700327] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:48 2019][2937630.708018] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:48 2019][2937630.714301] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:48 2019][2937630.720240] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:48 2019][2937630.726783] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:48 2019][2937630.732979] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.739697] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:48 2019][2937630.746676] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:48 2019][2937630.752564] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.759662] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.767415] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:48 2019][2937630.774684] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.782553] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.789522] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:48 2019][2937630.795967] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:48 2019][2937630.802021] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:48 2019][2937630.808502] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.816109] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:49 2019][2937630.821171] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937630.827447] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:49 2019][2937630.834065] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937630.840343] mdt01_046 S ffff8b8c24e41040 0 118272 2 0x00000080 [Fri Apr 12 11:11:49 2019][2937630.847705] Call Trace: [Fri Apr 12 11:11:49 2019][2937630.850339] [] schedule+0x29/0x70 [Fri Apr 12 11:11:49 2019][2937630.855530] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.862435] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:49 2019][2937630.868231] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.874718] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.882291] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:49 2019][2937630.887350] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937630.893625] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:49 2019][2937630.900250] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937630.906520] mdt01_047 S ffff8b8c24e42080 0 118273 2 0x00000080 [Fri Apr 12 11:11:49 2019][2937630.913876] Call Trace: [Fri Apr 12 11:11:49 2019][2937630.916516] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:49 2019][2937630.923435] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.931181] [] schedule+0x29/0x70 [Fri Apr 12 11:11:49 2019][2937630.936363] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.943450] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.950585] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:49 2019][2937630.957727] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:49 2019][2937630.963515] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.970862] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937630.978884] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:49 2019][2937630.985603] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:49 2019][2937630.992579] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:49 2019][2937630.999318] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.007368] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.014437] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:49 2019][2937631.021627] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.029037] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:11:49 2019][2937631.036127] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.042785] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.049872] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.057025] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:49 2019][2937631.063745] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:49 2019][2937631.070373] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:49 2019][2937631.077236] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.084131] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.091453] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.098017] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.105281] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.112985] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.119303] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.126397] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.134144] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.141404] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.149271] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.156242] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:49 2019][2937631.162702] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:49 2019][2937631.168749] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.175229] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.182804] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:49 2019][2937631.187863] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.194134] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:49 2019][2937631.200753] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.207026] mdt00_047 S ffff8b8c24e430c0 0 118274 2 0x00000080 [Fri Apr 12 11:11:49 2019][2937631.214398] Call Trace: [Fri Apr 12 11:11:49 2019][2937631.217031] [] schedule+0x29/0x70 [Fri Apr 12 11:11:49 2019][2937631.222216] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.229094] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:49 2019][2937631.234888] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.241368] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.248941] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:49 2019][2937631.254003] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.260277] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:49 2019][2937631.266898] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.273174] mdt01_048 S ffff8b8c24e44100 0 118275 2 0x00000080 [Fri Apr 12 11:11:49 2019][2937631.280558] Call Trace: [Fri Apr 12 11:11:49 2019][2937631.283205] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.290135] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.297897] [] schedule+0x29/0x70 [Fri Apr 12 11:11:49 2019][2937631.303079] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.310165] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.317250] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:49 2019][2937631.324391] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:49 2019][2937631.330182] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.337558] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.345645] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:49 2019][2937631.352367] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:49 2019][2937631.359357] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:49 2019][2937631.366106] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.374173] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:49 2019][2937631.381376] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.388536] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:11:49 2019][2937631.395721] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.402390] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.409525] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.416680] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:49 2019][2937631.423407] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:49 2019][2937631.430064] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.436953] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:49 2019][2937631.443777] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.450662] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.457989] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.464561] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.471889] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.479631] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.485961] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.493056] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.500821] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.508107] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.516017] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.523002] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:49 2019][2937631.529454] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:49 2019][2937631.535531] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.542032] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.549605] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:49 2019][2937631.554687] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.560978] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:49 2019][2937631.567619] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.573897] mdt01_049 S ffff8b8c24e45140 0 118276 2 0x00000080 [Fri Apr 12 11:11:49 2019][2937631.581260] Call Trace: [Fri Apr 12 11:11:49 2019][2937631.583904] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.590841] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.598622] [] schedule+0x29/0x70 [Fri Apr 12 11:11:49 2019][2937631.603846] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.610928] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.618039] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:49 2019][2937631.625200] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:49 2019][2937631.630991] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.638334] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.646376] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.654145] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:49 2019][2937631.661062] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.668845] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.676884] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:49 2019][2937631.683523] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:49 2019][2937631.689733] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:49 2019][2937631.696472] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:49 2019][2937631.703454] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:49 2019][2937631.709353] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.716451] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.724255] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:49 2019][2937631.731520] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.739437] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.746407] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:49 2019][2937631.752870] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:49 2019][2937631.758933] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.765448] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.773040] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:49 2019][2937631.778109] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.784386] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:49 2019][2937631.791031] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.797313] mdt02_031 S ffff8b8c24e46180 0 118277 2 0x00000080 [Fri Apr 12 11:11:49 2019][2937631.804686] Call Trace: [Fri Apr 12 11:11:49 2019][2937631.807330] [] schedule+0x29/0x70 [Fri Apr 12 11:11:49 2019][2937631.812527] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.819406] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:49 2019][2937631.825196] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.831691] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:49 2019][2937631.839266] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:49 2019][2937631.844350] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.850663] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:49 2019][2937631.857295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:49 2019][2937631.863571] mdt00_048 S ffff8b8bad568000 0 118278 2 0x00000080 [Fri Apr 12 11:11:50 2019][2937631.870994] Call Trace: [Fri Apr 12 11:11:50 2019][2937631.873670] [] schedule+0x29/0x70 [Fri Apr 12 11:11:50 2019][2937631.878867] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937631.885752] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:50 2019][2937631.891546] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937631.898041] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937631.905625] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:50 2019][2937631.910779] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937631.917258] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:50 2019][2937631.923884] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937631.930157] mdt01_050 S ffff8b8bad569040 0 118279 2 0x00000080 [Fri Apr 12 11:11:50 2019][2937631.937520] Call Trace: [Fri Apr 12 11:11:50 2019][2937631.940163] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:50 2019][2937631.947091] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937631.954878] [] schedule+0x29/0x70 [Fri Apr 12 11:11:50 2019][2937631.960072] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937631.967175] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937631.974260] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:50 2019][2937631.981410] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:50 2019][2937631.987202] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937631.994566] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.002668] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.010469] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:50 2019][2937632.017452] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.025281] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.033326] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:50 2019][2937632.039983] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:50 2019][2937632.046186] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.052903] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.059894] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:50 2019][2937632.065804] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.072901] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.080657] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.087924] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.095813] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.102780] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:50 2019][2937632.109226] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:50 2019][2937632.115353] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.121854] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.129440] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:50 2019][2937632.134508] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937632.140810] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:50 2019][2937632.147428] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937632.153703] mdt00_049 S ffff8b8bad56a080 0 118280 2 0x00000080 [Fri Apr 12 11:11:50 2019][2937632.161072] Call Trace: [Fri Apr 12 11:11:50 2019][2937632.163717] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.170648] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.178463] [] schedule+0x29/0x70 [Fri Apr 12 11:11:50 2019][2937632.183733] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.190904] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.197982] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:50 2019][2937632.205137] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:50 2019][2937632.210930] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.213734] LustreError: dumping log to /tmp/lustre-log.1555092710.118225 [Fri Apr 12 11:11:50 2019][2937632.225240] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.233260] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.239975] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:50 2019][2937632.246951] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.253717] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.261775] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.268885] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.276099] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.283519] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:11:50 2019][2937632.290623] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.297310] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.304404] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.311555] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:50 2019][2937632.318297] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:50 2019][2937632.325013] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.331923] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.338907] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.346307] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.352876] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.360149] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.367863] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.374220] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.381318] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.389094] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.396378] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.404274] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.411244] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:50 2019][2937632.417700] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:50 2019][2937632.423771] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.430292] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.437909] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:50 2019][2937632.442974] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937632.449265] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:50 2019][2937632.455891] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937632.462163] mdt01_051 S ffff8b8bad56b0c0 0 118281 2 0x00000080 [Fri Apr 12 11:11:50 2019][2937632.469536] Call Trace: [Fri Apr 12 11:11:50 2019][2937632.472186] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.479171] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.487010] [] schedule+0x29/0x70 [Fri Apr 12 11:11:50 2019][2937632.492293] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.499509] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.506629] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:50 2019][2937632.513838] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:50 2019][2937632.519622] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.526984] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.535062] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.541782] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:50 2019][2937632.548765] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.555510] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.563595] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.570693] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.577764] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.584925] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.591555] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:50 2019][2937632.598740] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.605406] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.612523] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.619690] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:50 2019][2937632.626448] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:50 2019][2937632.633096] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:50 2019][2937632.639949] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.646878] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.654236] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.660809] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.668078] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.675815] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.682149] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.689265] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.697082] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:50 2019][2937632.704366] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.712252] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.719219] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:50 2019][2937632.725664] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:50 2019][2937632.731717] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.738212] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:50 2019] kthread+0xd1/0xe0 [Fri Apr 12 11:11:50 2019][2937632.750855] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937632.757160] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:50 2019][2937632.763780] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937632.770053] mdt00_050 S ffff8b8bad56c100 0 118282 2 0x00000080 [Fri Apr 12 11:11:50 2019][2937632.777419] Call Trace: [Fri Apr 12 11:11:50 2019][2937632.780058] [] schedule+0x29/0x70 [Fri Apr 12 11:11:50 2019][2937632.785246] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.792129] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:50 2019][2937632.797934] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.804472] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.812110] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:50 2019][2937632.817180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937632.823463] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:50 2019][2937632.830109] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:50 2019][2937632.836392] mdt03_021 S ffff8b8bad56d140 0 118283 2 0x00000080 [Fri Apr 12 11:11:50 2019][2937632.843789] Call Trace: [Fri Apr 12 11:11:50 2019][2937632.846571] [] schedule+0x29/0x70 [Fri Apr 12 11:11:50 2019][2937632.851833] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.858721] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:50 2019][2937632.864515] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:50 2019][2937632.871010] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937632.878593] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:51 2019][2937632.883653] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937632.889929] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:51 2019][2937632.896584] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937632.902865] mdt02_032 S ffff8b8bad56e180 0 118284 2 0x00000080 [Fri Apr 12 11:11:51 2019][2937632.910238] Call Trace: [Fri Apr 12 11:11:51 2019][2937632.912881] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:51 2019][2937632.919806] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937632.927550] [] schedule+0x29/0x70 [Fri Apr 12 11:11:51 2019][2937632.932734] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937632.939866] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937632.946960] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:51 2019][2937632.954156] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:51 2019][2937632.959957] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937632.967313] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937632.975332] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:51 2019][2937632.982059] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:51 2019][2937632.989034] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:51 2019][2937632.995776] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.003822] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.010914] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.017995] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:51 2019][2937633.025177] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:51 2019][2937633.031808] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:51 2019][2937633.039000] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.045659] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.052780] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.059933] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:51 2019][2937633.066683] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:51 2019][2937633.073320] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:51 2019][2937633.080190] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.087083] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.094421] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.101004] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.108277] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.115987] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.122301] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.129394] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.137146] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.144406] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.152316] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.159287] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:51 2019][2937633.165734] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:51 2019][2937633.171821] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.178308] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.185883] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:51 2019][2937633.190966] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.197264] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:51 2019][2937633.203887] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.210186] mdt01_052 S ffff8b88eabf8000 0 118285 2 0x00000080 [Fri Apr 12 11:11:51 2019][2937633.217590] Call Trace: [Fri Apr 12 11:11:51 2019][2937633.220222] [] schedule+0x29/0x70 [Fri Apr 12 11:11:51 2019][2937633.225416] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.232295] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:51 2019][2937633.238084] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.244563] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.252212] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:51 2019][2937633.257298] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.263593] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:51 2019][2937633.270248] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.276520] mdt00_051 S ffff8b88eabf9040 0 118286 2 0x00000080 [Fri Apr 12 11:11:51 2019][2937633.283884] Call Trace: [Fri Apr 12 11:11:51 2019][2937633.286514] [] schedule+0x29/0x70 [Fri Apr 12 11:11:51 2019][2937633.291717] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.298614] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:51 2019][2937633.304416] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.310952] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.318538] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:51 2019][2937633.323600] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.329879] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:51 2019][2937633.336513] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.342788] mdt00_052 S ffff8b88eabfa080 0 118287 2 0x00000080 [Fri Apr 12 11:11:51 2019][2937633.350202] Call Trace: [Fri Apr 12 11:11:51 2019][2937633.352841] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.359759] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.367506] [] schedule+0x29/0x70 [Fri Apr 12 11:11:51 2019][2937633.372685] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.379765] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.386852] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:51 2019][2937633.393992] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:51 2019][2937633.399778] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.407119] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.415157] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:51 2019][2937633.421875] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:51 2019][2937633.428851] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:51 2019][2937633.435623] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.443709] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.450864] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.457951] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:51 2019][2937633.465112] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:51 2019][2937633.471743] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:51 2019][2937633.478923] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.485578] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.492684] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.499837] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:51 2019][2937633.506596] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:51 2019][2937633.513234] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:51 2019][2937633.520065] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.526954] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.534279] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.540864] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.548193] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.555904] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.562224] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.569331] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.577098] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.584376] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.592259] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.599235] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:51 2019][2937633.605713] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:51 2019][2937633.611770] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.618262] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.625848] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:51 2019][2937633.630912] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.637204] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:51 2019][2937633.643846] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.650125] mdt02_033 S ffff8b88eabfb0c0 0 118288 2 0x00000080 [Fri Apr 12 11:11:51 2019][2937633.657513] Call Trace: [Fri Apr 12 11:11:51 2019][2937633.660156] [] schedule+0x29/0x70 [Fri Apr 12 11:11:51 2019][2937633.665403] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.672298] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:51 2019][2937633.678113] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.684612] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.692202] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:51 2019][2937633.697262] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.703537] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:51 2019][2937633.710172] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:51 2019][2937633.716441] mdt01_053 S ffff8b88eabfc100 0 118290 2 0x00000080 [Fri Apr 12 11:11:51 2019][2937633.723812] Call Trace: [Fri Apr 12 11:11:51 2019][2937633.726445] [] schedule+0x29/0x70 [Fri Apr 12 11:11:51 2019][2937633.731589] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:11:51 2019][2937633.737648] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:51 2019][2937633.744971] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:11:51 2019][2937633.751538] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.758455] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.766251] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.773411] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.780478] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:51 2019][2937633.786265] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.793521] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:51 2019][2937633.800923] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.807766] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:51 2019][2937633.815805] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:51 2019][2937633.822436] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:51 2019][2937633.829188] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:51 2019][2937633.835603] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:51 2019][2937633.842676] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:51 2019][2937633.850377] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:52 2019][2937633.857704] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:52 2019][2937633.863995] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:52 2019][2937633.871407] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:52 2019][2937633.878300] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:52 2019][2937633.884738] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937633.891888] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:52 2019][2937633.898901] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:52 2019][2937633.905563] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:52 2019][2937633.913290] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:52 2019][2937633.919590] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:52 2019][2937633.925532] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:52 2019][2937633.932085] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:52 2019][2937633.938292] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:52 2019][2937633.945006] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:52 2019][2937633.952007] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:52 2019][2937633.957925] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937633.965020] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937633.972774] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:52 2019][2937633.980048] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937633.987922] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937633.994892] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:52 2019][2937634.001337] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:52 2019][2937634.007394] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.013875] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.021446] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:52 2019][2937634.026507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.032783] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:52 2019][2937634.039402] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.045677] mdt03_022 S ffff8b88eabfd140 0 118291 2 0x00000080 [Fri Apr 12 11:11:52 2019][2937634.053040] Call Trace: [Fri Apr 12 11:11:52 2019][2937634.055670] [] schedule+0x29/0x70 [Fri Apr 12 11:11:52 2019][2937634.060860] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.067735] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:52 2019][2937634.073526] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.080008] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.087583] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:52 2019][2937634.092642] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.098909] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:52 2019][2937634.105529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.111796] mdt00_053 S ffff8b88eabfe180 0 118292 2 0x00000080 [Fri Apr 12 11:11:52 2019][2937634.119152] Call Trace: [Fri Apr 12 11:11:52 2019][2937634.121782] [] schedule+0x29/0x70 [Fri Apr 12 11:11:52 2019][2937634.126967] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.133847] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:52 2019][2937634.139634] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.146119] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.153692] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:52 2019][2937634.158754] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.165029] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:52 2019][2937634.171655] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.177930] mdt02_034 S ffff8b886ca68000 0 118293 2 0x00000080 [Fri Apr 12 11:11:52 2019][2937634.185308] Call Trace: [Fri Apr 12 11:11:52 2019][2937634.187950] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:52 2019][2937634.195013] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.202805] [] schedule+0x29/0x70 [Fri Apr 12 11:11:52 2019][2937634.208006] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.215089] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.222206] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:52 2019][2937634.229385] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:52 2019][2937634.235190] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.242564] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.250592] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.257334] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:52 2019][2937634.264319] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.271065] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.279108] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.286210] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.293276] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.300433] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.307079] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:52 2019][2937634.314264] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.320936] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.328052] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.335219] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:52 2019][2937634.341939] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:52 2019][2937634.348576] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.355438] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.362322] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:52 2019][2937634.369649] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:52 2019][2937634.376229] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.383535] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.391239] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.397575] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.404677] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.412435] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:52 2019][2937634.419702] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.427608] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.434591] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:52 2019][2937634.441036] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:52 2019][2937634.447091] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.453580] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.461154] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:52 2019][2937634.466213] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.472499] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:52 2019][2937634.479124] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.485418] mdt00_054 S ffff8b886ca69040 0 118294 2 0x00000080 [Fri Apr 12 11:11:52 2019][2937634.492790] Call Trace: [Fri Apr 12 11:11:52 2019][2937634.495460] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.503222] [] schedule+0x29/0x70 [Fri Apr 12 11:11:52 2019][2937634.508447] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.515530] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.522586] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:52 2019][2937634.528372] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.535653] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:52 2019][2937634.543003] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.549849] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.557878] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:52 2019][2937634.564509] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:52 2019][2937634.571221] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:52 2019][2937634.577639] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:52 2019][2937634.584732] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:52 2019][2937634.592425] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:52 2019][2937634.599755] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:52 2019][2937634.606041] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:52 2019][2937634.613451] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:52 2019][2937634.620348] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.626755] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.633919] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.640897] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.647544] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:52 2019][2937634.655226] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:52 2019][2937634.661514] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:52 2019][2937634.667449] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:52 2019][2937634.673996] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:52 2019][2937634.680200] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.686918] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:52 2019][2937634.693902] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:52 2019][2937634.699801] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.706897] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.714646] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:52 2019][2937634.721916] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.729786] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.736770] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:52 2019][2937634.743215] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:52 2019][2937634.749269] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.755765] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.763338] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:52 2019][2937634.768398] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.774676] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:52 2019][2937634.781296] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.787569] mdt03_023 S ffff8b886ca6a080 0 118295 2 0x00000080 [Fri Apr 12 11:11:52 2019][2937634.794934] Call Trace: [Fri Apr 12 11:11:52 2019][2937634.797565] [] schedule+0x29/0x70 [Fri Apr 12 11:11:52 2019][2937634.802756] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.809634] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:52 2019][2937634.815421] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.821905] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:52 2019][2937634.829477] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:52 2019][2937634.834538] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.840811] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:52 2019][2937634.847432] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:52 2019][2937634.853704] mdt01_054 S ffff8b886ca6b0c0 0 118296 2 0x00000080 [Fri Apr 12 11:11:53 2019][2937634.861061] Call Trace: [Fri Apr 12 11:11:53 2019][2937634.863693] [] schedule+0x29/0x70 [Fri Apr 12 11:11:53 2019][2937634.868884] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937634.875768] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:53 2019][2937634.881556] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937634.888041] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937634.895611] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:53 2019][2937634.900672] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937634.906947] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:53 2019][2937634.913570] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937634.919844] mdt01_055 S ffff8b886ca6c100 0 118297 2 0x00000080 [Fri Apr 12 11:11:53 2019][2937634.927207] Call Trace: [Fri Apr 12 11:11:53 2019][2937634.929844] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:53 2019][2937634.936767] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937634.944518] [] schedule+0x29/0x70 [Fri Apr 12 11:11:53 2019][2937634.949704] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937634.956791] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937634.963870] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:53 2019][2937634.971014] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:53 2019][2937634.976804] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937634.984157] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937634.992191] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:53 2019][2937634.998906] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:53 2019][2937635.005884] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.012626] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.020690] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.027801] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.034878] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.042050] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.048683] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:53 2019][2937635.055882] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.062552] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.069656] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.076804] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:53 2019][2937635.083520] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:53 2019][2937635.090149] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.097010] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.103902] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:53 2019][2937635.111224] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:53 2019][2937635.117823] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.125087] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.132799] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.139134] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.146228] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.153976] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:53 2019][2937635.161268] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.169197] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.176196] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:53 2019][2937635.182655] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:53 2019][2937635.188881] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.195381] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.202968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:53 2019][2937635.208050] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.214377] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:53 2019][2937635.221005] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.227285] mdt03_024 S ffff8b886ca6d140 0 118298 2 0x00000080 [Fri Apr 12 11:11:53 2019][2937635.234674] Call Trace: [Fri Apr 12 11:11:53 2019][2937635.237307] [] schedule+0x29/0x70 [Fri Apr 12 11:11:53 2019][2937635.242508] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.249398] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:53 2019][2937635.255191] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.261678] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.269265] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:53 2019][2937635.274330] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.280615] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:53 2019][2937635.287247] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.293552] mdt00_055 S ffff8b886ca6e180 0 118299 2 0x00000080 [Fri Apr 12 11:11:53 2019][2937635.300922] Call Trace: [Fri Apr 12 11:11:53 2019][2937635.303558] [] schedule+0x29/0x70 [Fri Apr 12 11:11:53 2019][2937635.308761] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.315644] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:53 2019][2937635.321443] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.327946] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.335521] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:53 2019][2937635.340582] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.346855] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:53 2019][2937635.353493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.359774] mdt03_025 S ffff8b8a5db80000 0 118300 2 0x00000080 [Fri Apr 12 11:11:53 2019][2937635.367164] Call Trace: [Fri Apr 12 11:11:53 2019][2937635.369813] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:53 2019][2937635.376697] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:53 2019][2937635.382844] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.390623] [] schedule+0x29/0x70 [Fri Apr 12 11:11:53 2019][2937635.395809] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.402900] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.409983] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:53 2019][2937635.417135] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:53 2019][2937635.422954] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.430326] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.438365] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.446129] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.452604] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.460387] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.468409] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.475056] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:53 2019][2937635.482751] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:53 2019][2937635.489054] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:53 2019][2937635.494989] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:53 2019][2937635.501535] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:53 2019][2937635.507751] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.514472] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.521450] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:53 2019][2937635.527341] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.534437] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.542192] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:53 2019][2937635.549451] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.557319] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.564299] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:53 2019][2937635.570747] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:53 2019][2937635.576818] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.583322] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.590893] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:53 2019][2937635.595968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.602252] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:53 2019][2937635.608895] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.615172] mdt02_035 S ffff8b77566e2400 0 118301 2 0x00000080 [Fri Apr 12 11:11:53 2019][2937635.622543] Call Trace: [Fri Apr 12 11:11:53 2019][2937635.625175] [] schedule+0x29/0x70 [Fri Apr 12 11:11:53 2019][2937635.630378] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.637282] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:53 2019][2937635.643080] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.649564] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.657146] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:53 2019][2937635.662221] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.668517] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:53 2019][2937635.675146] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:53 2019][2937635.681419] mdt01_056 S ffff8b8a5db82080 0 118302 2 0x00000080 [Fri Apr 12 11:11:53 2019][2937635.688792] Call Trace: [Fri Apr 12 11:11:53 2019][2937635.691430] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:53 2019][2937635.698361] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.706122] [] schedule+0x29/0x70 [Fri Apr 12 11:11:53 2019][2937635.711305] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.718406] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.725504] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:53 2019][2937635.732644] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:53 2019][2937635.738446] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.745846] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.753867] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.760587] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:53 2019][2937635.767576] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.774348] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.782369] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.789605] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.796759] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:11:53 2019][2937635.803955] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.810737] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.817837] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.824990] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:53 2019][2937635.831734] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:53 2019][2937635.838361] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:53 2019][2937635.845193] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:53 2019][2937635.852092] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:53 2019][2937635.859430] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:54 2019][2937635.865998] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.873262] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.880967] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.887283] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.894376] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.902128] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:54 2019][2937635.909396] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.917266] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.924234] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:54 2019][2937635.930682] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:54 2019][2937635.936778] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.943263] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.950842] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:54 2019][2937635.955937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937635.962228] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:54 2019][2937635.968850] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937635.975125] mdt00_056 S ffff8b8a5db830c0 0 118303 2 0x00000080 [Fri Apr 12 11:11:54 2019][2937635.982492] Call Trace: [Fri Apr 12 11:11:54 2019][2937635.985135] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:54 2019][2937635.992054] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937635.999816] [] schedule+0x29/0x70 [Fri Apr 12 11:11:54 2019][2937636.005009] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.012103] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.019207] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:54 2019][2937636.026347] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:54 2019][2937636.032137] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.039481] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.047502] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.054220] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:54 2019][2937636.061195] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.067937] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.075984] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.083124] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.090206] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.097364] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.103991] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:54 2019][2937636.111170] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.117826] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.124917] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.132064] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:54 2019][2937636.138783] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:54 2019][2937636.145409] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.152236] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.159129] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.166449] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.173022] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.180288] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.187993] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.194306] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.201404] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.209160] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.216427] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.224296] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.231262] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:54 2019][2937636.237709] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:54 2019][2937636.243830] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.250315] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.257892] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:54 2019][2937636.262960] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937636.269246] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:54 2019][2937636.275864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937636.282136] mdt02_036 S ffff8b8a5db84100 0 118304 2 0x00000080 [Fri Apr 12 11:11:54 2019][2937636.289498] Call Trace: [Fri Apr 12 11:11:54 2019][2937636.292141] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.299061] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.306823] [] schedule+0x29/0x70 [Fri Apr 12 11:11:54 2019][2937636.312015] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.319102] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.326185] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:54 2019][2937636.333327] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:54 2019][2937636.339118] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.346464] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.354505] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.362275] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:54 2019][2937636.369200] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.376978] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.385009] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:54 2019][2937636.391640] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:54 2019][2937636.397840] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.404570] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.411542] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:54 2019][2937636.417440] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.424551] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.432310] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.439602] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.447501] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.454472] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:54 2019][2937636.460924] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:54 2019][2937636.466982] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.473469] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.481065] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:54 2019][2937636.486130] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937636.492445] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:54 2019][2937636.499078] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937636.505360] mdt00_057 S ffff8b8a5db85140 0 118305 2 0x00000080 [Fri Apr 12 11:11:54 2019][2937636.512734] Call Trace: [Fri Apr 12 11:11:54 2019][2937636.515381] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.522305] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.530064] [] schedule+0x29/0x70 [Fri Apr 12 11:11:54 2019][2937636.535254] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.542353] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.549448] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:54 2019][2937636.556593] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:54 2019][2937636.562381] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.569734] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.577753] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.584487] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:54 2019][2937636.591470] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.598210] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.606228] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.613413] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.620564] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:11:54 2019][2937636.627741] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.634398] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.641494] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.648648] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:54 2019][2937636.655371] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:54 2019][2937636.661999] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:54 2019][2937636.668843] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.675743] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.683065] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.689659] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.696940] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.704674] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.710996] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.718087] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.725844] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:54 2019][2937636.733110] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.740980] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.747946] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:54 2019][2937636.754392] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:54 2019][2937636.760440] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.766922] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.774492] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:54 2019][2937636.779553] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937636.785830] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:54 2019][2937636.792451] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937636.798722] mdt01_057 S ffff8b8a5db86180 0 118306 2 0x00000080 [Fri Apr 12 11:11:54 2019][2937636.806080] Call Trace: [Fri Apr 12 11:11:54 2019][2937636.808711] [] schedule+0x29/0x70 [Fri Apr 12 11:11:54 2019][2937636.813970] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.820889] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:54 2019][2937636.826680] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.833162] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:54 2019][2937636.840750] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:54 2019][2937636.845813] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937636.852098] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:54 2019][2937636.858718] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:54 2019][2937636.864989] mdt02_037 S ffff8b8b8e4b8000 0 118307 2 0x00000080 [Fri Apr 12 11:11:55 2019][2937636.872347] Call Trace: [Fri Apr 12 11:11:55 2019][2937636.875124] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937636.882912] [] schedule+0x29/0x70 [Fri Apr 12 11:11:55 2019][2937636.888106] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937636.895208] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937636.902274] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:55 2019][2937636.908061] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937636.915318] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:55 2019][2937636.922684] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937636.929508] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937636.937532] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:55 2019][2937636.944176] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:55 2019][2937636.950893] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:55 2019][2937636.957283] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:55 2019][2937636.964338] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:55 2019][2937636.972031] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:55 2019][2937636.979370] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:55 2019][2937636.985657] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:55 2019][2937636.993076] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:55 2019][2937636.999966] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.006382] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.013536] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.020514] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.027187] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.034907] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.041202] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:55 2019][2937637.047138] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:55 2019][2937637.053709] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:55 2019][2937637.059907] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.066630] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.073607] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:55 2019][2937637.079523] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.086634] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.094396] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:55 2019][2937637.101669] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.109542] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.116517] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:55 2019][2937637.122976] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:55 2019][2937637.129030] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.135519] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.143105] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:55 2019][2937637.148177] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:55 2019][2937637.154450] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:55 2019][2937637.161083] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:55 2019][2937637.167351] mdt00_058 S ffff8b8b8e4b9040 0 118308 2 0x00000080 [Fri Apr 12 11:11:55 2019][2937637.174718] Call Trace: [Fri Apr 12 11:11:55 2019][2937637.177365] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:55 2019][2937637.184303] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.192060] [] schedule+0x29/0x70 [Fri Apr 12 11:11:55 2019][2937637.197268] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.204348] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.211427] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.218570] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:55 2019][2937637.224377] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.231735] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.239797] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.247636] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:55 2019][2937637.254596] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.262396] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.270446] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:55 2019][2937637.277111] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:55 2019][2937637.283329] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.290077] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.297062] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:55 2019][2937637.302980] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.310090] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.317864] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:55 2019][2937637.325144] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.332983] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:55 2019][2937637.339454] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:55 2019][2937637.345550] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.352041] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.359622] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:55 2019][2937637.364681] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:55 2019][2937637.370959] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:55 2019][2937637.377579] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:55 2019][2937637.383852] mdt01_058 S ffff8b8b8e4ba080 0 118309 2 0x00000080 [Fri Apr 12 11:11:55 2019][2937637.391224] Call Trace: [Fri Apr 12 11:11:55 2019][2937637.393863] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:55 2019][2937637.400784] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.408558] [] schedule+0x29/0x70 [Fri Apr 12 11:11:55 2019][2937637.413748] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.420834] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.427916] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.435059] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:55 2019][2937637.440852] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.448233] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.456280] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.464073] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:11:55 2019][2937637.470992] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.478767] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.486783] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:11:55 2019][2937637.493417] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:55 2019][2937637.499617] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.506344] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.513367] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:55 2019][2937637.519270] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.526368] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.534160] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:55 2019][2937637.541430] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.549304] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.556293] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:55 2019][2937637.562763] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:55 2019][2937637.568842] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.575336] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.582934] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:55 2019][2937637.587991] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:55 2019][2937637.594271] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:55 2019][2937637.600898] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:55 2019][2937637.607180] mdt01_059 S ffff8b8b8e4bb0c0 0 118310 2 0x00000080 [Fri Apr 12 11:11:55 2019][2937637.614602] Call Trace: [Fri Apr 12 11:11:55 2019][2937637.617278] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.625064] [] schedule+0x29/0x70 [Fri Apr 12 11:11:55 2019][2937637.630247] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.637338] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.644397] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:55 2019][2937637.650183] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.657439] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.664787] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.671616] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.679639] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:55 2019][2937637.686263] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:55 2019][2937637.692982] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:55 2019][2937637.699367] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.706429] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:55 2019][2937637.714111] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.721434] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:55 2019][2937637.727723] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:55 2019][2937637.735132] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:55 2019][2937637.742025] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.748424] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.755579] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.762555] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.769210] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.776889] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:55 2019][2937637.783172] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:55 2019][2937637.789107] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:55 2019][2937637.795653] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:55 2019][2937637.801847] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.808565] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:55 2019][2937637.815540] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:55 2019][2937637.821432] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.828526] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.836278] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:55 2019][2937637.843546] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.851414] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:55 2019][2937637.858380] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:55 2019][2937637.864827] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:55 2019][2937637.870884] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937637.877372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937637.884944] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937637.890005] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937637.896280] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937637.902901] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937637.909175] mdt00_059 S ffff8b8b8e4bc100 0 118311 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937637.916567] Call Trace: [Fri Apr 12 11:11:56 2019][2937637.919210] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937637.924400] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937637.931316] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937637.937119] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937637.943625] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937637.951203] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937637.956263] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937637.962553] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937637.969178] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937637.975460] mdt02_038 S ffff8b8b8e4bd140 0 118312 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937637.982832] Call Trace: [Fri Apr 12 11:11:56 2019][2937637.985463] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937637.990609] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:11:56 2019][2937637.996655] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:56 2019][2937638.003987] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:11:56 2019][2937638.010544] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Fri Apr 12 11:11:56 2019][2937638.017468] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.025255] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.032342] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.039418] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937638.045202] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.052459] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:56 2019][2937638.059829] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.066670] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.074696] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:56 2019][2937638.081347] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:56 2019][2937638.088126] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:56 2019][2937638.094537] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:56 2019][2937638.101600] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:56 2019][2937638.109294] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:56 2019][2937638.116617] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:56 2019][2937638.122925] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:56 2019][2937638.130341] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:56 2019][2937638.137251] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.143685] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.150867] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.157851] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.164498] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:56 2019][2937638.172180] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:56 2019][2937638.178469] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:56 2019][2937638.184404] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:56 2019][2937638.190967] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:56 2019][2937638.197209] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.203934] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.210915] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:56 2019][2937638.216814] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.223909] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.231657] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:56 2019][2937638.238929] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.246794] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.253775] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:56 2019][2937638.260229] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:56 2019][2937638.266287] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.272802] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.280383] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937638.285460] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.291743] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937638.298386] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.304662] mdt01_060 S ffff8b8b8e4be180 0 118313 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937638.312042] Call Trace: [Fri Apr 12 11:11:56 2019][2937638.314674] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937638.319866] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.326745] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937638.332539] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.339039] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.346623] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937638.351682] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.357954] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937638.364576] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.370848] mdt00_060 S ffff8b869a658000 0 118314 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937638.378205] Call Trace: [Fri Apr 12 11:11:56 2019][2937638.380839] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937638.386029] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.392929] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937638.398722] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.405227] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.412806] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937638.417880] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.424157] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937638.430777] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.437049] mdt01_061 S ffff8b869a659040 0 118315 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937638.444420] Call Trace: [Fri Apr 12 11:11:56 2019][2937638.447054] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937638.452244] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.460318] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.467385] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937638.473173] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.479830] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.487401] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937638.492627] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.498897] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937638.505688] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.511963] mdt03_026 S ffff8b869a65a080 0 118316 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937638.519319] Call Trace: [Fri Apr 12 11:11:56 2019][2937638.521962] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937638.527151] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.534051] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937638.539861] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.546346] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.553924] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937638.559002] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.565289] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937638.571908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.578199] mdt01_062 R running task 0 118317 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937638.585603] Call Trace: [Fri Apr 12 11:11:56 2019][2937638.588240] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937638.593443] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.600422] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937638.606224] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.612891] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.620466] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937638.625699] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.631976] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937638.638770] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.645042] mdt00_061 S ffff8b869a65c100 0 118318 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937638.652399] Call Trace: [Fri Apr 12 11:11:56 2019][2937638.655029] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937638.660262] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.667149] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937638.672936] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.679419] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.687007] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:56 2019][2937638.692082] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.698358] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:56 2019][2937638.704978] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:56 2019][2937638.711254] mdt02_039 S ffff8b8cf92da080 0 118319 2 0x00000080 [Fri Apr 12 11:11:56 2019][2937638.718613] Call Trace: [Fri Apr 12 11:11:56 2019][2937638.721254] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:56 2019][2937638.728169] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.735938] [] schedule+0x29/0x70 [Fri Apr 12 11:11:56 2019][2937638.741118] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.748214] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.755300] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:56 2019][2937638.762452] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:56 2019][2937638.768239] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.775604] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.783648] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.790383] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:56 2019][2937638.797374] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.804125] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.812182] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.819304] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.826369] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.833519] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:56 2019][2937638.840146] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:56 2019][2937638.847326] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.853981] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.861067] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:56 2019][2937638.868217] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:56 2019][2937638.874938] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:57 2019][2937638.881565] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:57 2019][2937638.888388] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.895274] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:57 2019][2937638.902616] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:57 2019][2937638.909208] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.916499] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.924217] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.930528] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.937666] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.945449] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:57 2019][2937638.952735] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.960608] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.967579] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:57 2019][2937638.974040] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:57 2019][2937638.980112] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.986614] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937638.994212] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:57 2019][2937638.999291] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:57 2019][2937639.005565] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:57 2019][2937639.012194] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:57 2019][2937639.018474] mdt01_063 S ffff8b8b70a4e180 0 118321 2 0x00000080 [Fri Apr 12 11:11:57 2019][2937639.025863] Call Trace: [Fri Apr 12 11:11:57 2019][2937639.028505] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:57 2019][2937639.035396] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:57 2019][2937639.041533] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.049285] [] schedule+0x29/0x70 [Fri Apr 12 11:11:57 2019][2937639.054470] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.061553] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.068633] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.075771] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:57 2019][2937639.081559] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.088908] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.096950] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.104706] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.111181] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.118952] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.126969] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.133614] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.141298] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.147580] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:57 2019][2937639.153515] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:57 2019][2937639.160088] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:57 2019][2937639.166290] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.173016] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.179990] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:57 2019][2937639.185900] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.193035] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.200811] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:57 2019][2937639.208087] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.215959] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.222929] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:57 2019][2937639.229373] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:57 2019][2937639.235431] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.241915] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.249492] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:57 2019][2937639.254561] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:57 2019][2937639.260846] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:57 2019][2937639.267466] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:57 2019][2937639.273737] mdt01_064 S ffff8b8c307a8000 0 118323 2 0x00000080 [Fri Apr 12 11:11:57 2019][2937639.281103] Call Trace: [Fri Apr 12 11:11:57 2019][2937639.283739] [] schedule+0x29/0x70 [Fri Apr 12 11:11:57 2019][2937639.288937] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.295816] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:57 2019][2937639.301606] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.308088] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.315662] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:57 2019][2937639.320731] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:57 2019][2937639.327025] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:57 2019][2937639.333646] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:57 2019][2937639.339918] mdt01_065 S ffff8b8c307ad140 0 118324 2 0x00000080 [Fri Apr 12 11:11:57 2019][2937639.347283] Call Trace: [Fri Apr 12 11:11:57 2019][2937639.349953] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.357710] [] schedule+0x29/0x70 [Fri Apr 12 11:11:57 2019][2937639.362915] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.370017] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.377078] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:57 2019][2937639.382871] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.390138] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.397523] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.404357] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.412591] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:57 2019][2937639.419248] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:57 2019][2937639.425993] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:57 2019][2937639.432395] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.439502] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:57 2019][2937639.447196] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.454531] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:57 2019][2937639.460820] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:57 2019][2937639.468257] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:57 2019][2937639.475160] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.481566] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.488715] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.495691] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.502341] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.510023] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.516321] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:57 2019][2937639.522256] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:57 2019][2937639.528802] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:57 2019][2937639.535006] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.541731] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.548710] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:57 2019][2937639.554614] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.561739] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.569496] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:57 2019][2937639.576764] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.584631] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.591613] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:57 2019][2937639.598064] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:57 2019][2937639.604114] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.610605] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.618179] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:57 2019][2937639.623238] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:57 2019][2937639.629514] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:57 2019][2937639.636141] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:57 2019][2937639.642433] mdt01_066 S ffff8b8c307ac100 0 118326 2 0x00000080 [Fri Apr 12 11:11:57 2019][2937639.649797] Call Trace: [Fri Apr 12 11:11:57 2019][2937639.652440] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:57 2019][2937639.659322] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:11:57 2019][2937639.665489] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.673240] [] schedule+0x29/0x70 [Fri Apr 12 11:11:57 2019][2937639.678417] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.685502] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.692586] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.699726] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:57 2019][2937639.705514] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.712863] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.720904] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.728661] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.735159] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.742954] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.751006] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.757667] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.765348] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:57 2019][2937639.771636] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:57 2019][2937639.777573] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:57 2019][2937639.784117] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:57 2019][2937639.790320] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.797060] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:57 2019][2937639.804043] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:57 2019][2937639.809938] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.817040] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.824815] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:57 2019][2937639.832083] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.839993] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:57 2019][2937639.846963] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:57 2019][2937639.853407] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:58 2019][2937639.859483] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937639.865970] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937639.873555] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:58 2019][2937639.878621] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937639.884897] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:58 2019][2937639.891516] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937639.897786] mdt01_067 S ffff8b8c307a9040 0 118327 2 0x00000080 [Fri Apr 12 11:11:58 2019][2937639.905140] Call Trace: [Fri Apr 12 11:11:58 2019][2937639.907782] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:58 2019][2937639.914705] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937639.922467] [] schedule+0x29/0x70 [Fri Apr 12 11:11:58 2019][2937639.927660] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937639.934764] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937639.941888] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:58 2019][2937639.949066] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:58 2019][2937639.954859] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937639.962228] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937639.970259] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:58 2019][2937639.976982] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:58 2019][2937639.983967] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:58 2019][2937639.990721] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937639.998798] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.005880] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.013083] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.020518] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:11:58 2019][2937640.027634] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.034299] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.041391] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.048543] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:58 2019][2937640.055259] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:58 2019][2937640.061914] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.068826] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.075685] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.082595] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:58 2019][2937640.089925] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:58 2019][2937640.096498] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.103772] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.111480] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.117797] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.124905] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.132661] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:58 2019][2937640.139947] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.147850] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.154821] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:58 2019][2937640.161273] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:58 2019][2937640.167343] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.173832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.181414] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:58 2019][2937640.186471] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937640.192741] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:58 2019][2937640.199358] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937640.205625] mdt01_068 S ffff8b8c307ae180 0 118329 2 0x00000080 [Fri Apr 12 11:11:58 2019][2937640.212989] Call Trace: [Fri Apr 12 11:11:58 2019][2937640.215658] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.223403] [] schedule+0x29/0x70 [Fri Apr 12 11:11:58 2019][2937640.228584] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.235671] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.242723] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:58 2019][2937640.248532] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.255786] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.263133] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.269984] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.278015] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:58 2019][2937640.284645] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:58 2019][2937640.291370] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:58 2019][2937640.297763] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.304829] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:58 2019][2937640.312514] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.319841] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:58 2019][2937640.326147] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:58 2019][2937640.333572] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:58 2019][2937640.340460] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.346886] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.354089] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.361097] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.367779] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.375461] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.381760] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:58 2019][2937640.387706] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:58 2019][2937640.394256] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:58 2019][2937640.400463] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.407187] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.414188] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:58 2019][2937640.420087] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.427186] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.434953] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:58 2019][2937640.442226] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.450108] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.457081] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:58 2019][2937640.462871] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.469358] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.476937] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:58 2019][2937640.482003] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937640.488297] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:58 2019][2937640.494916] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937640.501190] mdt01_069 S ffff8b4e29b81040 0 118331 2 0x00000080 [Fri Apr 12 11:11:58 2019][2937640.508546] Call Trace: [Fri Apr 12 11:11:58 2019][2937640.511179] [] schedule+0x29/0x70 [Fri Apr 12 11:11:58 2019][2937640.516367] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.523249] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:58 2019][2937640.529043] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.535526] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.543098] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:58 2019][2937640.548157] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937640.554432] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:58 2019][2937640.561054] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937640.567328] mdt01_070 S ffff8b4e29b830c0 0 118332 2 0x00000080 [Fri Apr 12 11:11:58 2019][2937640.574691] Call Trace: [Fri Apr 12 11:11:58 2019][2937640.577359] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.585106] [] schedule+0x29/0x70 [Fri Apr 12 11:11:58 2019][2937640.590287] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.597374] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.604431] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:58 2019][2937640.610222] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.617476] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.624822] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.631643] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.639657] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:58 2019][2937640.646283] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:58 2019][2937640.653003] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:58 2019][2937640.659386] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.666441] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:58 2019][2937640.674128] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.681443] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:58 2019][2937640.687725] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:58 2019][2937640.695134] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:58 2019][2937640.702024] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.708424] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.715591] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.722564] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.729211] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.736893] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:58 2019][2937640.743174] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:58 2019][2937640.749110] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:58 2019][2937640.755656] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:58 2019][2937640.761851] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.768569] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:58 2019][2937640.775546] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:58 2019][2937640.781434] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.788530] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.796280] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:58 2019][2937640.803546] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.811416] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.818384] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:58 2019][2937640.824175] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.830657] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:58 2019][2937640.838231] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:58 2019][2937640.843292] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:58 2019][2937640.849566] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:58 2019][2937640.856188] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937640.862461] mdt01_071 S ffff8b4e29b84100 0 118333 2 0x00000080 [Fri Apr 12 11:11:59 2019][2937640.869825] Call Trace: [Fri Apr 12 11:11:59 2019][2937640.872461] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:11:59 2019][2937640.879381] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937640.887129] [] schedule+0x29/0x70 [Fri Apr 12 11:11:59 2019][2937640.892311] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937640.899397] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937640.906475] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:11:59 2019][2937640.913615] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:59 2019][2937640.919409] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937640.926754] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937640.934772] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:59 2019][2937640.941490] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:11:59 2019][2937640.948464] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:11:59 2019][2937640.955205] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937640.963251] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937640.970345] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937640.977406] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:11:59 2019][2937640.984572] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:11:59 2019][2937640.991203] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:11:59 2019][2937640.998381] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.005035] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.012122] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.019273] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:11:59 2019][2937641.025991] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:11:59 2019][2937641.032620] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.039447] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.046336] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:11:59 2019][2937641.053658] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:11:59 2019][2937641.060225] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.067491] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.075192] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.081508] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.088604] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.096353] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:59 2019][2937641.103612] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.111481] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.118449] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:59 2019][2937641.124893] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:59 2019][2937641.130947] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.137427] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.145003] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:59 2019][2937641.150063] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937641.156338] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:59 2019][2937641.162959] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937641.169234] mdt01_072 S ffff8b6cf9652080 0 118334 2 0x00000080 [Fri Apr 12 11:11:59 2019][2937641.176615] Call Trace: [Fri Apr 12 11:11:59 2019][2937641.179246] [] schedule+0x29/0x70 [Fri Apr 12 11:11:59 2019][2937641.184433] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.191311] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:59 2019][2937641.197100] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.203581] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.211155] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:59 2019][2937641.216216] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937641.222492] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:59 2019][2937641.229112] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937641.235386] mdt01_073 S ffff8b783475b0c0 0 118335 2 0x00000080 [Fri Apr 12 11:11:59 2019][2937641.242743] Call Trace: [Fri Apr 12 11:11:59 2019][2937641.245410] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.253157] [] schedule+0x29/0x70 [Fri Apr 12 11:11:59 2019][2937641.258336] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.265417] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.272475] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:59 2019][2937641.278263] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.285517] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.292870] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.299695] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.307707] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:59 2019][2937641.314334] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:59 2019][2937641.321053] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:59 2019][2937641.327441] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.334501] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:59 2019][2937641.342187] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.349505] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:59 2019][2937641.355792] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:59 2019][2937641.363202] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:59 2019][2937641.370094] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.376491] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.383653] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.390630] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.397279] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.404964] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.411250] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:59 2019][2937641.417187] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:59 2019][2937641.423732] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:59 2019][2937641.429927] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.436646] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.443623] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:59 2019][2937641.449510] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.456608] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.464355] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:59 2019][2937641.471615] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.479486] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.486451] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:11:59 2019][2937641.492898] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:11:59 2019][2937641.498951] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.505431] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.513005] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:59 2019][2937641.518067] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937641.524342] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:59 2019][2937641.530961] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937641.537228] mdt02_040 S ffff8b7834759040 0 118336 2 0x00000080 [Fri Apr 12 11:11:59 2019][2937641.544593] Call Trace: [Fri Apr 12 11:11:59 2019][2937641.547224] [] schedule+0x29/0x70 [Fri Apr 12 11:11:59 2019][2937641.552410] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.559287] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:11:59 2019][2937641.566255] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:59 2019][2937641.572047] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.578531] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.586102] [] kthread+0xd1/0xe0 [Fri Apr 12 11:11:59 2019][2937641.591162] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937641.597431] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:11:59 2019][2937641.604051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:11:59 2019][2937641.610325] mdt01_074 S ffff8b783475e180 0 118337 2 0x00000080 [Fri Apr 12 11:11:59 2019][2937641.617687] Call Trace: [Fri Apr 12 11:11:59 2019][2937641.620358] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.628103] [] schedule+0x29/0x70 [Fri Apr 12 11:11:59 2019][2937641.633285] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.640386] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.647438] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:11:59 2019][2937641.653228] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.660480] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.667828] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.674650] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.682669] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:11:59 2019][2937641.689298] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:11:59 2019][2937641.696014] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:11:59 2019][2937641.702403] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.709463] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:11:59 2019][2937641.717144] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.724468] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:11:59 2019][2937641.730754] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:11:59 2019][2937641.738166] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:11:59 2019][2937641.745055] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.751455] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.758600] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.765578] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.772226] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.779925] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:11:59 2019][2937641.786205] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:11:59 2019][2937641.792141] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:11:59 2019][2937641.798687] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:11:59 2019][2937641.804882] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.811598] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:11:59 2019][2937641.818575] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:11:59 2019][2937641.824465] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.831555] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.839312] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:11:59 2019][2937641.846593] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.854466] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:11:59 2019][2937641.861432] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:00 2019][2937641.867881] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:00 2019][2937641.873932] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937641.880415] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937641.887988] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:00 2019][2937641.893046] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937641.899312] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:00 2019][2937641.905935] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937641.912209] mdt01_075 S ffff8b6cf9655140 0 118338 2 0x00000080 [Fri Apr 12 11:12:00 2019][2937641.919571] Call Trace: [Fri Apr 12 11:12:00 2019][2937641.922245] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937641.929994] [] schedule+0x29/0x70 [Fri Apr 12 11:12:00 2019][2937641.935177] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937641.942265] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937641.949316] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:00 2019][2937641.955102] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937641.962332] [] ? kfree+0x106/0x140 [Fri Apr 12 11:12:00 2019][2937641.967601] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937641.974428] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937641.982447] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:00 2019][2937641.989074] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:00 2019][2937641.995784] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:00 2019][2937642.002172] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:00 2019][2937642.009232] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.016916] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:00 2019][2937642.024238] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:00 2019][2937642.030525] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:00 2019][2937642.037937] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:00 2019][2937642.044828] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.051233] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.058380] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.065356] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.072005] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:00 2019][2937642.079688] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:00 2019][2937642.085973] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:00 2019][2937642.091910] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:00 2019][2937642.098457] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:00 2019][2937642.104650] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.111369] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.118343] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:00 2019][2937642.124236] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.131331] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.139083] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.146350] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.154217] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.161185] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:00 2019][2937642.166975] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.173458] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.181032] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:00 2019][2937642.186091] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937642.192367] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:00 2019][2937642.198986] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937642.205261] mdt01_076 S ffff8b6cf9650000 0 118339 2 0x00000080 [Fri Apr 12 11:12:00 2019][2937642.212624] Call Trace: [Fri Apr 12 11:12:00 2019][2937642.215261] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.222170] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.229922] [] schedule+0x29/0x70 [Fri Apr 12 11:12:00 2019][2937642.235102] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.242190] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.249273] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:00 2019][2937642.256415] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:00 2019][2937642.262202] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.269544] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.277563] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.284278] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:00 2019][2937642.291256] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.297995] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.306034] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.313128] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.320197] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.327345] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.333978] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:00 2019][2937642.341155] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.347811] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.354898] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.362048] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:00 2019][2937642.368767] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:00 2019][2937642.375398] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.382221] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.389111] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.396435] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.403010] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.410274] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.417977] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.424290] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.431376] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.439126] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.446388] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.454271] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.461242] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:00 2019][2937642.467030] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.473514] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.481086] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:00 2019][2937642.486145] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937642.492422] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:00 2019][2937642.499041] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937642.505307] mdt01_077 S ffff8b6cf9654100 0 118340 2 0x00000080 [Fri Apr 12 11:12:00 2019][2937642.512664] Call Trace: [Fri Apr 12 11:12:00 2019][2937642.515299] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.522211] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.529960] [] schedule+0x29/0x70 [Fri Apr 12 11:12:00 2019][2937642.535142] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.542229] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.549311] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:00 2019][2937642.556453] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:00 2019][2937642.562242] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.569591] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.577636] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.585416] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:00 2019][2937642.592326] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.600108] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.608128] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:00 2019][2937642.614758] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:00 2019][2937642.620955] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.627675] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:00 2019][2937642.634657] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:00 2019][2937642.640545] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.647631] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.655383] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.662643] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.670512] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.677478] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:00 2019][2937642.683924] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:00 2019][2937642.689979] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.696460] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.704033] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:00 2019][2937642.709092] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937642.715361] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:00 2019][2937642.721982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937642.728256] mdt01_078 S ffff8b6cf9651040 0 118341 2 0x00000080 [Fri Apr 12 11:12:00 2019][2937642.735617] Call Trace: [Fri Apr 12 11:12:00 2019][2937642.738250] [] schedule+0x29/0x70 [Fri Apr 12 11:12:00 2019][2937642.743439] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.750322] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:00 2019][2937642.756115] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.762598] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.770169] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:00 2019][2937642.775230] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937642.781522] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:00 2019][2937642.788144] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:00 2019][2937642.794418] mdt01_079 S ffff8b783475c100 0 118342 2 0x00000080 [Fri Apr 12 11:12:00 2019][2937642.801782] Call Trace: [Fri Apr 12 11:12:00 2019][2937642.804418] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:00 2019][2937642.811340] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.819087] [] schedule+0x29/0x70 [Fri Apr 12 11:12:00 2019][2937642.824265] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.831346] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.838420] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:00 2019][2937642.845562] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:00 2019][2937642.851350] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:00 2019][2937642.858701] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.866748] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.874510] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:01 2019][2937642.881427] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.889208] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.897229] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:01 2019][2937642.903859] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:01 2019][2937642.910057] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:01 2019][2937642.916773] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:01 2019][2937642.923748] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:01 2019][2937642.929636] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.936727] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.944484] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:01 2019][2937642.951743] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.959612] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.966579] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:01 2019][2937642.973023] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:01 2019][2937642.979073] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.985577] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937642.993150] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:01 2019][2937642.998212] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.004486] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:01 2019][2937643.011108] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.017381] mdt03_027 S ffff8b8cd09ad140 0 118343 2 0x00000080 [Fri Apr 12 11:12:01 2019][2937643.024736] Call Trace: [Fri Apr 12 11:12:01 2019][2937643.027367] [] schedule+0x29/0x70 [Fri Apr 12 11:12:01 2019][2937643.032556] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.039431] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:01 2019][2937643.045224] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.051705] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.059278] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:01 2019][2937643.064340] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.070614] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:01 2019][2937643.077235] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.083510] mdt01_080 S ffff8b8cd09ac100 0 118344 2 0x00000080 [Fri Apr 12 11:12:01 2019][2937643.090873] Call Trace: [Fri Apr 12 11:12:01 2019][2937643.093510] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:01 2019][2937643.100428] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.108190] [] schedule+0x29/0x70 [Fri Apr 12 11:12:01 2019][2937643.113367] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.120457] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.127542] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:01 2019][2937643.134679] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:01 2019][2937643.140471] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.147835] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.155855] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:01 2019][2937643.162572] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:01 2019][2937643.169549] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:01 2019][2937643.176292] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.184340] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.191440] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.198510] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:01 2019][2937643.205665] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:01 2019][2937643.212296] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:01 2019][2937643.219473] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.226126] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.233216] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.240365] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:01 2019][2937643.247084] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:01 2019][2937643.253712] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:01 2019][2937643.260539] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.267429] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:01 2019][2937643.274753] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:01 2019][2937643.281322] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.288591] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.296292] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.302608] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.309704] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.317463] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:01 2019][2937643.324731] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.332597] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.339567] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:01 2019][2937643.346013] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:01 2019][2937643.352066] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.358544] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.366120] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:01 2019][2937643.371180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.377457] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:01 2019][2937643.384077] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.390353] mdt01_081 S ffff8b8cd09a9040 0 118345 2 0x00000080 [Fri Apr 12 11:12:01 2019][2937643.397716] Call Trace: [Fri Apr 12 11:12:01 2019][2937643.400345] [] schedule+0x29/0x70 [Fri Apr 12 11:12:01 2019][2937643.405536] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.412418] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:01 2019][2937643.418211] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.424694] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.432264] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:01 2019][2937643.437328] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.443617] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:01 2019][2937643.450232] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.456505] mdt02_041 S ffff8b8cd09a8000 0 118347 2 0x00000080 [Fri Apr 12 11:12:01 2019][2937643.463868] Call Trace: [Fri Apr 12 11:12:01 2019][2937643.466501] [] schedule+0x29/0x70 [Fri Apr 12 11:12:01 2019][2937643.471686] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.478592] [] ? keys_fini+0xb1/0x1d0 [obdclass] [Fri Apr 12 11:12:01 2019][2937643.485040] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:01 2019][2937643.490831] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.497314] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.504885] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:01 2019][2937643.509946] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.516221] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:01 2019][2937643.522844] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.529117] mdt03_028 S ffff8b8cd09ae180 0 118348 2 0x00000080 [Fri Apr 12 11:12:01 2019][2937643.536479] Call Trace: [Fri Apr 12 11:12:01 2019][2937643.539110] [] schedule+0x29/0x70 [Fri Apr 12 11:12:01 2019][2937643.544300] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.551184] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:01 2019][2937643.556978] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.563457] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.571031] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:01 2019][2937643.576091] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.582366] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:01 2019][2937643.588989] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.595261] mdt01_082 S ffff8b8cd09aa080 0 118349 2 0x00000080 [Fri Apr 12 11:12:01 2019][2937643.602617] Call Trace: [Fri Apr 12 11:12:01 2019][2937643.605248] [] schedule+0x29/0x70 [Fri Apr 12 11:12:01 2019][2937643.610437] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.617320] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:01 2019][2937643.623111] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.629594] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.637168] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:01 2019][2937643.642227] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.648505] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:01 2019][2937643.655125] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.661399] mdt01_083 S ffff8b8cd09ab0c0 0 118350 2 0x00000080 [Fri Apr 12 11:12:01 2019][2937643.668763] Call Trace: [Fri Apr 12 11:12:01 2019][2937643.671405] [] ? mdt_stripe_get+0xd6/0x400 [mdt] [Fri Apr 12 11:12:01 2019][2937643.677851] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:12:01 2019][2937643.683264] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Apr 12 11:12:01 2019][2937643.691183] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Apr 12 11:12:01 2019][2937643.698065] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Apr 12 11:12:01 2019][2937643.705119] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Fri Apr 12 11:12:01 2019][2937643.711833] [] ? lnet_ni_send+0x44/0xd0 [lnet] [Fri Apr 12 11:12:01 2019][2937643.718118] [] ? lnet_send+0x82/0x1c0 [lnet] [Fri Apr 12 11:12:01 2019][2937643.724226] [] ? LNetPut+0x2cc/0xb50 [lnet] [Fri Apr 12 11:12:01 2019][2937643.730280] [] ? ptl_send_buf+0x146/0x530 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.736902] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:01 2019][2937643.744126] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:12:01 2019][2937643.749793] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:12:01 2019][2937643.755548] [] ? account_entity_dequeue+0xae/0xd0 [Fri Apr 12 11:12:01 2019][2937643.762084] [] ? dequeue_entity+0x11c/0x5e0 [Fri Apr 12 11:12:01 2019][2937643.768098] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:12:01 2019][2937643.774373] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:12:01 2019][2937643.780302] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:12:01 2019][2937643.786056] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:12:01 2019][2937643.792675] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:12:01 2019][2937643.798866] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:01 2019][2937643.804243] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.811307] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:01 2019][2937643.817097] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.823751] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:01 2019][2937643.831323] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:01 2019][2937643.836555] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.842824] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:01 2019][2937643.849616] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:01 2019][2937643.855884] mdt01_084 S ffff8b7da96030c0 0 118351 2 0x00000080 [Fri Apr 12 11:12:01 2019][2937643.863248] Call Trace: [Fri Apr 12 11:12:01 2019][2937643.865916] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937643.873669] [] schedule+0x29/0x70 [Fri Apr 12 11:12:02 2019][2937643.878854] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937643.885942] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937643.892996] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:02 2019][2937643.898786] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937643.906039] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:02 2019][2937643.913391] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937643.920219] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937643.928239] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:02 2019][2937643.934864] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:02 2019][2937643.941575] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:02 2019][2937643.947963] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:02 2019][2937643.955024] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:02 2019][2937643.962710] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:02 2019][2937643.970049] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:02 2019][2937643.976336] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:02 2019][2937643.983771] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:02 2019][2937643.990669] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:02 2019][2937643.997075] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.004233] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.011217] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.017866] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:02 2019][2937644.025547] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:02 2019][2937644.031834] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:02 2019][2937644.037780] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:02 2019][2937644.044328] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:02 2019][2937644.050541] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.057265] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.064282] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:02 2019][2937644.070198] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.077293] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.085053] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.092340] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.100216] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.107184] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:02 2019][2937644.113639] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:02 2019][2937644.119687] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.126170] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.133743] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:02 2019][2937644.138803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:02 2019][2937644.145074] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:02 2019][2937644.151729] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:02 2019][2937644.158003] mdt01_085 S ffff8b8cc4ef8000 0 118352 2 0x00000080 [Fri Apr 12 11:12:02 2019][2937644.165365] Call Trace: [Fri Apr 12 11:12:02 2019][2937644.168007] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.174923] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.182677] [] schedule+0x29/0x70 [Fri Apr 12 11:12:02 2019][2937644.187859] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.194948] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.202029] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:02 2019][2937644.209173] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:02 2019][2937644.214961] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.222307] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.230322] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.237039] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:02 2019][2937644.244013] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.250754] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.258775] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.265993] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.273184] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:12:02 2019][2937644.280368] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.287020] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.294107] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.301259] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:02 2019][2937644.307977] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:02 2019][2937644.314627] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.321511] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.328337] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.335222] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.342581] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.349152] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.356427] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.364158] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.370469] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.377569] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.385329] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.392596] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.400490] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.407464] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:02 2019][2937644.413910] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:02 2019][2937644.419989] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.426500] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.434081] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:02 2019][2937644.439146] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:02 2019][2937644.445445] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:02 2019][2937644.452074] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:02 2019][2937644.458344] mdt01_086 S ffff8b8cc4ef9040 0 118353 2 0x00000080 [Fri Apr 12 11:12:02 2019][2937644.465755] Call Trace: [Fri Apr 12 11:12:02 2019][2937644.468401] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.475348] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.483101] [] schedule+0x29/0x70 [Fri Apr 12 11:12:02 2019][2937644.488288] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.495403] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.502497] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:02 2019][2937644.509648] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:02 2019][2937644.515445] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.522829] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.530856] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.537573] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:02 2019][2937644.544546] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.551291] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.559345] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.566434] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.573547] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.580708] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.587337] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:02 2019][2937644.594539] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.601198] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.608301] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.615459] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:02 2019][2937644.622185] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:02 2019][2937644.628825] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.635654] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.642541] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.649868] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.656448] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.663713] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.671443] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.677787] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.684928] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.692692] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.699978] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.707870] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.714838] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:02 2019][2937644.721282] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:02 2019][2937644.727338] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.733846] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.741433] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:02 2019][2937644.746495] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:02 2019][2937644.752770] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:02 2019][2937644.759389] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:02 2019][2937644.765662] mdt01_087 S ffff8b8cc4efa080 0 118354 2 0x00000080 [Fri Apr 12 11:12:02 2019][2937644.773025] Call Trace: [Fri Apr 12 11:12:02 2019][2937644.775675] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:02 2019][2937644.782593] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.790341] [] schedule+0x29/0x70 [Fri Apr 12 11:12:02 2019][2937644.795564] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.802662] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.809756] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:02 2019][2937644.816897] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:02 2019][2937644.822687] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.830035] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.838055] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.844787] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:02 2019][2937644.851762] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:02 2019][2937644.858523] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:02 2019][2937644.866596] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.873695] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.880768] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:03 2019][2937644.887923] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:03 2019][2937644.894550] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:03 2019][2937644.901735] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.908391] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.915482] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.922630] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:03 2019][2937644.929347] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:03 2019][2937644.935977] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:03 2019][2937644.942804] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.949696] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:03 2019][2937644.957015] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:03 2019][2937644.963582] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.970847] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.978550] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.984865] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.991962] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937644.999716] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.006981] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.014853] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.021823] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:03 2019][2937645.028299] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:03 2019][2937645.034443] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.040945] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.048523] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:03 2019][2937645.053585] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:03 2019][2937645.059861] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:03 2019][2937645.066482] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:03 2019][2937645.072756] mdt01_088 S ffff8b8cc4efb0c0 0 118355 2 0x00000080 [Fri Apr 12 11:12:03 2019][2937645.080147] Call Trace: [Fri Apr 12 11:12:03 2019][2937645.082837] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.090598] [] schedule+0x29/0x70 [Fri Apr 12 11:12:03 2019][2937645.095791] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.102892] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.109952] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:03 2019][2937645.115743] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.122974] [] ? kfree+0x106/0x140 [Fri Apr 12 11:12:03 2019][2937645.128243] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.135071] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.143090] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:03 2019][2937645.149715] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:03 2019][2937645.156426] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:03 2019][2937645.162816] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:03 2019][2937645.169876] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.177560] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:03 2019][2937645.184876] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:03 2019][2937645.191156] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:03 2019][2937645.198567] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:03 2019][2937645.205457] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.211860] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.219014] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.225989] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.232642] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:03 2019][2937645.240328] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:03 2019][2937645.246616] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:03 2019][2937645.252554] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:03 2019][2937645.259097] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:03 2019][2937645.265291] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.272011] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.278986] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:03 2019][2937645.284885] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.291984] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.299739] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.307073] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.314961] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.321950] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:03 2019][2937645.328403] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:03 2019][2937645.334482] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.340969] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.348548] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:03 2019][2937645.353628] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:03 2019][2937645.359966] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:03 2019][2937645.366599] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:03 2019][2937645.372901] mdt01_089 S ffff8b8cc4efc100 0 118356 2 0x00000080 [Fri Apr 12 11:12:03 2019][2937645.380299] Call Trace: [Fri Apr 12 11:12:03 2019][2937645.382944] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.389902] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.397673] [] schedule+0x29/0x70 [Fri Apr 12 11:12:03 2019][2937645.402861] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.409946] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.417030] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:03 2019][2937645.424176] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:03 2019][2937645.429976] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.437325] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.445338] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.452056] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:03 2019][2937645.459030] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.465771] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.473823] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.480901] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.488109] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.495544] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:12:03 2019][2937645.502642] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.509353] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.516440] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.523593] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:03 2019][2937645.530308] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:03 2019][2937645.536938] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.543760] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.550646] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.557970] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.564541] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.571809] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.579520] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.585856] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.592948] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.600721] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.607981] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.615851] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.622821] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:03 2019][2937645.629294] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:03 2019][2937645.635383] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.641893] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.649470] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:03 2019][2937645.654530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:03 2019][2937645.660802] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:03 2019][2937645.667422] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:03 2019][2937645.673689] mdt03_029 S ffff8b8cc4efd140 0 118357 2 0x00000080 [Fri Apr 12 11:12:03 2019][2937645.681053] Call Trace: [Fri Apr 12 11:12:03 2019][2937645.683704] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.690587] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:03 2019][2937645.696725] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.704508] [] schedule+0x29/0x70 [Fri Apr 12 11:12:03 2019][2937645.709699] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.716810] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.723922] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:03 2019][2937645.731076] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:03 2019][2937645.736876] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.744238] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.752294] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.760100] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.766606] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.774407] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.782438] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.789099] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:03 2019][2937645.796815] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:03 2019][2937645.803100] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:03 2019][2937645.809044] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:03 2019][2937645.815624] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:03 2019][2937645.821828] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.828563] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:03 2019][2937645.835602] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:03 2019][2937645.841514] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.848610] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.856402] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:03 2019][2937645.863683] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:03 2019][2937645.871558] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937645.878527] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:04 2019][2937645.884973] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:04 2019][2937645.891029] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937645.897520] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937645.905107] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:04 2019][2937645.910170] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937645.916461] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:04 2019][2937645.923082] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937645.929353] mdt03_030 S ffff8b8cc4efe180 0 118358 2 0x00000080 [Fri Apr 12 11:12:04 2019][2937645.936715] Call Trace: [Fri Apr 12 11:12:04 2019][2937645.939388] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937645.947140] [] schedule+0x29/0x70 [Fri Apr 12 11:12:04 2019][2937645.952322] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937645.959426] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937645.966477] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:04 2019][2937645.972269] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937645.979551] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:04 2019][2937645.986925] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937645.993767] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.001817] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:04 2019][2937646.008467] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:04 2019][2937646.015196] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:04 2019][2937646.021584] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:04 2019][2937646.028648] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:04 2019][2937646.036326] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:04 2019][2937646.043644] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:04 2019][2937646.049930] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:04 2019][2937646.057341] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:04 2019][2937646.064251] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.070657] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.077822] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.084799] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.091468] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:04 2019][2937646.099185] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:04 2019][2937646.105478] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:04 2019][2937646.111412] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:04 2019][2937646.117985] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:04 2019][2937646.124195] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.130917] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.137925] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:04 2019][2937646.143828] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.150923] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.158689] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:04 2019][2937646.165983] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.173860] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.180829] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:04 2019][2937646.187275] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:04 2019][2937646.193337] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.199835] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.207408] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:04 2019][2937646.212465] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937646.218742] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:04 2019][2937646.225362] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937646.231636] mdt02_042 S ffff8b8842e50000 0 118359 2 0x00000080 [Fri Apr 12 11:12:04 2019][2937646.238999] Call Trace: [Fri Apr 12 11:12:04 2019][2937646.241637] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:04 2019][2937646.248557] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.256323] [] schedule+0x29/0x70 [Fri Apr 12 11:12:04 2019][2937646.261512] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.268610] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.275722] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:04 2019][2937646.283417] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:04 2019][2937646.289234] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.296589] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.304632] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.312415] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:04 2019][2937646.319340] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.327117] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.335136] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:04 2019][2937646.341773] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:04 2019][2937646.347968] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.354719] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.361697] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:04 2019][2937646.367629] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.374730] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.382490] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:04 2019][2937646.389788] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.397672] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.404643] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:04 2019][2937646.411112] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:04 2019][2937646.417176] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.423673] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.431246] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:04 2019][2937646.436316] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937646.442617] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:04 2019][2937646.449265] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937646.455547] mdt03_031 S ffff8b8842e51040 0 118360 2 0x00000080 [Fri Apr 12 11:12:04 2019][2937646.462918] Call Trace: [Fri Apr 12 11:12:04 2019][2937646.465548] [] schedule+0x29/0x70 [Fri Apr 12 11:12:04 2019][2937646.470742] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.477622] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:04 2019][2937646.483432] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.489915] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.497504] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:04 2019][2937646.502573] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937646.508847] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:04 2019][2937646.515469] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937646.521740] mdt00_062 S ffff8b8842e52080 0 118361 2 0x00000080 [Fri Apr 12 11:12:04 2019][2937646.529096] Call Trace: [Fri Apr 12 11:12:04 2019][2937646.531739] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:04 2019][2937646.538684] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.546448] [] schedule+0x29/0x70 [Fri Apr 12 11:12:04 2019][2937646.551662] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.558751] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.565845] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:04 2019][2937646.572985] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:04 2019][2937646.578768] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.586122] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.594211] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.601978] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:04 2019][2937646.608959] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.616752] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.624802] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:04 2019][2937646.631457] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:04 2019][2937646.637656] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.644374] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:04 2019][2937646.651351] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:04 2019][2937646.657242] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.664335] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.672086] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:04 2019][2937646.679354] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.687221] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.694188] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:04 2019][2937646.700635] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:04 2019][2937646.706691] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.713171] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.720743] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:04 2019][2937646.725804] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937646.732081] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:04 2019][2937646.738700] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:04 2019][2937646.744973] mdt02_043 S ffff8b8842e530c0 0 118362 2 0x00000080 [Fri Apr 12 11:12:04 2019][2937646.752336] Call Trace: [Fri Apr 12 11:12:04 2019][2937646.755008] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.762760] [] schedule+0x29/0x70 [Fri Apr 12 11:12:04 2019][2937646.767943] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.775027] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.782079] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:04 2019][2937646.787866] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.795123] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:04 2019][2937646.802472] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.809298] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:04 2019][2937646.817310] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:04 2019][2937646.823937] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:04 2019][2937646.830664] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:04 2019][2937646.837051] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:04 2019][2937646.844113] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:04 2019][2937646.851791] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:05 2019][2937646.859127] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:05 2019][2937646.865414] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:05 2019][2937646.872826] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:05 2019][2937646.879715] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:05 2019][2937646.886113] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937646.893258] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:05 2019][2937646.900235] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:05 2019][2937646.906886] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:05 2019][2937646.914568] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:05 2019][2937646.920853] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:05 2019][2937646.926790] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:05 2019][2937646.933337] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:05 2019][2937646.939530] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:05 2019][2937646.946247] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:05 2019][2937646.953223] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:05 2019][2937646.959115] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937646.966213] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937646.973969] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:05 2019][2937646.981238] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937646.989104] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937646.996073] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:05 2019][2937647.002520] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:05 2019][2937647.008575] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.015066] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.022636] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:05 2019][2937647.027696] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.033973] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:05 2019][2937647.040593] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.046866] mdt00_063 S ffff8b8842e54100 0 118363 2 0x00000080 [Fri Apr 12 11:12:05 2019][2937647.054249] Call Trace: [Fri Apr 12 11:12:05 2019][2937647.056879] [] schedule+0x29/0x70 [Fri Apr 12 11:12:05 2019][2937647.062070] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.068951] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:05 2019][2937647.074745] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.081226] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.088798] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:05 2019][2937647.093873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.100143] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:05 2019][2937647.106764] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.113037] mdt02_044 S ffff8b8842e55140 0 118364 2 0x00000080 [Fri Apr 12 11:12:05 2019][2937647.120393] Call Trace: [Fri Apr 12 11:12:05 2019][2937647.123031] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:05 2019][2937647.129951] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.137698] [] schedule+0x29/0x70 [Fri Apr 12 11:12:05 2019][2937647.142880] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.149967] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.157059] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:05 2019][2937647.164201] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:05 2019][2937647.169993] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.177337] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.185358] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:05 2019][2937647.192076] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:05 2019][2937647.199052] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:05 2019][2937647.205794] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.213844] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.220913] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:05 2019][2937647.228091] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.235505] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:12:05 2019][2937647.242598] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.249255] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.256339] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.263489] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:05 2019][2937647.270210] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:05 2019][2937647.276834] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:05 2019][2937647.283661] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.290544] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:05 2019][2937647.297866] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:05 2019][2937647.304432] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.311697] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.319402] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.325717] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.332809] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.340561] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:05 2019][2937647.347827] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.355696] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.362665] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:05 2019][2937647.369111] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:05 2019][2937647.375164] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.381644] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.389218] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:05 2019][2937647.394278] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.400556] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:05 2019][2937647.407176] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.413450] mdt02_045 S ffff8b8842e56180 0 118365 2 0x00000080 [Fri Apr 12 11:12:05 2019][2937647.420812] Call Trace: [Fri Apr 12 11:12:05 2019][2937647.423444] [] schedule+0x29/0x70 [Fri Apr 12 11:12:05 2019][2937647.428629] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.435509] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:05 2019][2937647.441307] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.447789] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.455363] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:05 2019][2937647.460421] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.466691] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:05 2019][2937647.473314] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.479586] mdt01_090 S ffff8b853a6e0000 0 118366 2 0x00000080 [Fri Apr 12 11:12:05 2019][2937647.486951] Call Trace: [Fri Apr 12 11:12:05 2019][2937647.489586] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:05 2019][2937647.496463] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:05 2019][2937647.502600] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.510348] [] schedule+0x29/0x70 [Fri Apr 12 11:12:05 2019][2937647.515530] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.522616] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.529698] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:05 2019][2937647.536840] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:05 2019][2937647.542626] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.549969] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.558017] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.565772] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:05 2019][2937647.572253] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.580039] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.588057] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:05 2019][2937647.594702] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:05 2019][2937647.602391] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:05 2019][2937647.608675] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:05 2019][2937647.614612] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:05 2019][2937647.621158] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:05 2019][2937647.627366] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:05 2019][2937647.634086] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:05 2019][2937647.641064] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:05 2019][2937647.646950] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.654046] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.661797] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:05 2019][2937647.669059] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.676926] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.683891] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:05 2019][2937647.690339] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:05 2019][2937647.696390] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.702874] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.710445] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:05 2019][2937647.715509] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.721785] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:05 2019][2937647.728403] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.734678] mdt02_046 S ffff8b853a6e1040 0 118367 2 0x00000080 [Fri Apr 12 11:12:05 2019][2937647.742042] Call Trace: [Fri Apr 12 11:12:05 2019][2937647.744673] [] schedule+0x29/0x70 [Fri Apr 12 11:12:05 2019][2937647.749859] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.756752] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:05 2019][2937647.762546] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.769030] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.776600] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:05 2019][2937647.781663] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.787937] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:05 2019][2937647.794557] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:05 2019][2937647.800830] mdt01_091 S ffff8b853a6e2080 0 118368 2 0x00000080 [Fri Apr 12 11:12:05 2019][2937647.808187] Call Trace: [Fri Apr 12 11:12:05 2019][2937647.810822] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:05 2019][2937647.817734] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.825483] [] schedule+0x29/0x70 [Fri Apr 12 11:12:05 2019][2937647.830664] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.837752] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:05 2019][2937647.844826] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:05 2019][2937647.851969] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:05 2019][2937647.857756] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.865098] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.873140] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.880908] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:06 2019][2937647.887823] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.895607] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.903627] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:06 2019][2937647.910257] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:06 2019][2937647.916452] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:06 2019][2937647.923170] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:06 2019][2937647.930146] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:06 2019][2937647.936037] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.943130] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.950881] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:06 2019][2937647.958138] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.966013] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.972983] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:06 2019][2937647.978778] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.985258] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937647.992830] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:06 2019][2937647.997892] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:06 2019][2937648.004165] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:06 2019][2937648.010788] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:06 2019][2937648.017061] mdt00_064 S ffff8b853a6e30c0 0 118369 2 0x00000080 [Fri Apr 12 11:12:06 2019][2937648.024418] Call Trace: [Fri Apr 12 11:12:06 2019][2937648.027053] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:06 2019][2937648.033965] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.041711] [] schedule+0x29/0x70 [Fri Apr 12 11:12:06 2019][2937648.046896] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.053984] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.061065] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:06 2019][2937648.068206] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:06 2019][2937648.073996] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.081344] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.089401] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.097160] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:06 2019][2937648.104079] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.111859] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.119874] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:06 2019][2937648.126524] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:06 2019][2937648.132716] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.139434] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.146410] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:06 2019][2937648.152297] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.159402] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.167155] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:06 2019][2937648.174422] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.182294] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.189259] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:06 2019][2937648.195056] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.201543] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.209115] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:06 2019][2937648.214171] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:06 2019][2937648.220440] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:06 2019][2937648.227075] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:06 2019][2937648.233344] mdt01_092 S ffff8b853a6e4100 0 118370 2 0x00000080 [Fri Apr 12 11:12:06 2019][2937648.240706] Call Trace: [Fri Apr 12 11:12:06 2019][2937648.243343] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:06 2019][2937648.250256] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.258002] [] schedule+0x29/0x70 [Fri Apr 12 11:12:06 2019][2937648.263183] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.270270] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.277345] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:06 2019][2937648.284479] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:06 2019][2937648.290281] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.297626] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.305645] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.312362] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:06 2019][2937648.319337] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.326080] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.334136] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.341230] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.348298] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.355447] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.362077] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:06 2019][2937648.369254] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.375906] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.382999] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.390147] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:06 2019][2937648.396867] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:06 2019][2937648.403516] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.410401] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.417228] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.424118] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:06 2019][2937648.431454] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:06 2019][2937648.438029] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.445296] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.453001] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.459315] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.466409] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.474160] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:06 2019][2937648.481418] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.489286] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.496256] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:06 2019][2937648.502715] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:06 2019][2937648.508762] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.515246] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.522819] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:06 2019][2937648.527877] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:06 2019][2937648.534146] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:06 2019][2937648.540766] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:06 2019][2937648.547040] mdt01_093 S ffff8b853a6e5140 0 118371 2 0x00000080 [Fri Apr 12 11:12:06 2019][2937648.554396] Call Trace: [Fri Apr 12 11:12:06 2019][2937648.557031] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:06 2019][2937648.563940] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.571682] [] schedule+0x29/0x70 [Fri Apr 12 11:12:06 2019][2937648.576862] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.583943] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.591029] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:06 2019][2937648.598167] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:06 2019][2937648.603957] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.611307] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.619350] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.627114] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:06 2019][2937648.634047] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.641827] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.649843] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:06 2019][2937648.656475] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:06 2019][2937648.662669] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.669387] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:06 2019][2937648.676360] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:06 2019][2937648.682250] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.689343] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.697136] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:06 2019][2937648.704499] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.712451] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.719421] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:06 2019][2937648.725866] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:06 2019][2937648.731916] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.738398] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.745976] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:06 2019][2937648.751038] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:06 2019][2937648.757309] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:06 2019][2937648.763941] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:06 2019][2937648.770246] mdt03_032 S ffff8b853a6e6180 0 118373 2 0x00000080 [Fri Apr 12 11:12:06 2019][2937648.777648] Call Trace: [Fri Apr 12 11:12:06 2019][2937648.780322] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.788071] [] schedule+0x29/0x70 [Fri Apr 12 11:12:06 2019][2937648.793251] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.800337] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.807396] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:06 2019][2937648.813181] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.820432] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:06 2019][2937648.827775] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.834612] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:06 2019][2937648.842629] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:06 2019][2937648.849254] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:06 2019][2937648.855974] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:06 2019][2937648.862359] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:07 2019][2937648.869422] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:07 2019][2937648.877109] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:07 2019][2937648.884479] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:07 2019][2937648.890784] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:07 2019][2937648.898201] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:07 2019][2937648.905092] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:07 2019][2937648.911505] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937648.918665] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:07 2019][2937648.925659] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:07 2019][2937648.932318] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:07 2019][2937648.940007] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:07 2019][2937648.946294] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:07 2019][2937648.952264] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:07 2019][2937648.958809] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:07 2019][2937648.965006] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:07 2019][2937648.971750] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:07 2019][2937648.978732] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:07 2019][2937648.984632] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937648.991740] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937648.999496] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:07 2019][2937649.006774] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.014650] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.021673] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:07 2019][2937649.028124] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:07 2019][2937649.034189] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.040708] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.048295] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:07 2019][2937649.053363] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.059647] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:07 2019][2937649.066267] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.072539] mdt02_047 S ffff8b8b64ed8000 0 118374 2 0x00000080 [Fri Apr 12 11:12:07 2019][2937649.079928] Call Trace: [Fri Apr 12 11:12:07 2019][2937649.082576] [] schedule+0x29/0x70 [Fri Apr 12 11:12:07 2019][2937649.087769] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.094653] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:07 2019][2937649.100453] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.106941] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.114513] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:07 2019][2937649.119575] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.125851] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:07 2019][2937649.132473] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.138746] mdt01_094 R running task 0 118376 2 0x00000080 [Fri Apr 12 11:12:07 2019][2937649.146111] Call Trace: [Fri Apr 12 11:12:07 2019][2937649.148814] [] ? tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.156048] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:07 2019][2937649.161426] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.168489] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:07 2019][2937649.174312] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.180972] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.188547] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:07 2019][2937649.193781] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.200054] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:07 2019][2937649.206848] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.213116] mdt02_048 S ffff8b5e11b4b0c0 0 118379 2 0x00000080 [Fri Apr 12 11:12:07 2019][2937649.220479] Call Trace: [Fri Apr 12 11:12:07 2019][2937649.223145] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.230892] [] schedule+0x29/0x70 [Fri Apr 12 11:12:07 2019][2937649.236073] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.243152] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.250203] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:07 2019][2937649.255988] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.263245] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:07 2019][2937649.270592] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.277414] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.285428] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:07 2019][2937649.292069] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:07 2019][2937649.298781] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:07 2019][2937649.305165] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:07 2019][2937649.312221] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:07 2019][2937649.319905] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:07 2019][2937649.327224] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:07 2019][2937649.333510] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:07 2019][2937649.340921] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:07 2019][2937649.347815] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.354214] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.361364] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.368360] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.375012] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:07 2019][2937649.382746] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:07 2019][2937649.389057] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:07 2019][2937649.395050] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:07 2019][2937649.401628] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:07 2019][2937649.407830] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.414546] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.421520] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:07 2019][2937649.427435] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.434525] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.442275] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:07 2019][2937649.449530] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.457413] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.464397] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:07 2019][2937649.470845] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:07 2019][2937649.476895] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.483375] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.490950] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:07 2019][2937649.496011] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.502286] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:07 2019][2937649.508906] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.515172] mdt00_065 S ffff8b5e11b48000 0 118380 2 0x00000080 [Fri Apr 12 11:12:07 2019][2937649.522537] Call Trace: [Fri Apr 12 11:12:07 2019][2937649.525173] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:07 2019][2937649.532047] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:07 2019][2937649.538185] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.545932] [] schedule+0x29/0x70 [Fri Apr 12 11:12:07 2019][2937649.551114] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.558221] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.565305] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:07 2019][2937649.572446] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:07 2019][2937649.578233] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.585580] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.593615] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.601371] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.607847] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.615629] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.623653] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.630301] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:07 2019][2937649.637984] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:07 2019][2937649.644271] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:07 2019][2937649.650206] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:07 2019][2937649.656749] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:07 2019][2937649.662945] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.669665] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:07 2019][2937649.676642] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:07 2019][2937649.682531] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.689627] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.697382] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:07 2019][2937649.704644] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.712515] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.719479] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:07 2019][2937649.725926] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:07 2019][2937649.731978] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.738459] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.746032] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:07 2019][2937649.751094] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.757369] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:07 2019][2937649.763990] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.770264] mdt03_033 S ffff8b5e11b4a080 0 118381 2 0x00000080 [Fri Apr 12 11:12:07 2019][2937649.777621] Call Trace: [Fri Apr 12 11:12:07 2019][2937649.780250] [] schedule+0x29/0x70 [Fri Apr 12 11:12:07 2019][2937649.785438] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.792315] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:07 2019][2937649.798106] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.804588] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:07 2019][2937649.812161] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:07 2019][2937649.817223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.823514] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:07 2019][2937649.830137] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:07 2019][2937649.836411] mdt01_095 S ffff8b5e11b49040 0 118382 2 0x00000080 [Fri Apr 12 11:12:07 2019][2937649.843774] Call Trace: [Fri Apr 12 11:12:07 2019][2937649.846410] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:07 2019][2937649.853296] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:07 2019][2937649.859433] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937649.867179] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937649.872361] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937649.879447] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937649.886523] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:08 2019][2937649.893665] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937649.899453] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937649.906803] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937649.914844] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937649.922598] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:08 2019][2937649.929079] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937649.936859] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937649.944882] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:08 2019][2937649.951527] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:08 2019][2937649.959226] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:08 2019][2937649.965505] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:08 2019][2937649.971446] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:08 2019][2937649.977989] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:08 2019][2937649.984186] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:08 2019][2937649.990903] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:08 2019][2937649.997879] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:08 2019][2937650.003765] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.010857] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.018613] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:08 2019][2937650.025871] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.033742] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.040709] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:08 2019][2937650.047157] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:08 2019][2937650.053209] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.059692] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.067264] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.072324] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.078600] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.085220] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.091508] mdt01_096 S ffff8b8b64ede180 0 118383 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.098864] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.101496] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.106682] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.113560] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.119352] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.125832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.133406] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.138467] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.144743] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.151364] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.157637] mdt00_066 S ffff8b8b64edd140 0 118384 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.164993] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.167622] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.172799] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.179680] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.185470] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.191954] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.199529] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.204588] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.210862] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.217486] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.223756] mdt03_034 S ffff8b8b64edc100 0 118385 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.231112] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.233745] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.238930] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.245807] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.251600] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.258080] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.265654] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.270717] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.276991] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.283612] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.289886] mdt00_067 S ffff8b8b64edb0c0 0 118386 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.297249] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.299879] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.305068] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.311953] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.317744] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.324226] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.331816] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.336869] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.343146] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.349767] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.356041] mdt01_097 S ffff8b8b64eda080 0 118387 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.363404] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.366035] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.371222] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.378101] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.383893] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.390385] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.397957] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.403015] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.409348] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.415984] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.422255] mdt01_098 S ffff8b8a7b2f0000 0 118389 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.429621] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.432269] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.437459] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.444406] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.450204] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.456684] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.464294] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.469356] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.475631] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.482255] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.488525] mdt02_049 S ffff8b8a7b2f1040 0 118390 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.495885] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.498517] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.503717] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.510610] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.516402] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.522889] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.530466] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.535534] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.541812] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.548430] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.554731] mdt01_099 S ffff8b8a7b2f2080 0 118391 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.562406] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.565060] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:08 2019][2937650.571990] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.579754] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.584943] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.592031] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.599135] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:08 2019][2937650.606283] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.612074] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.619425] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.627480] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.635245] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:08 2019][2937650.642157] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.649934] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.657955] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:08 2019][2937650.664588] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:08 2019][2937650.670783] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:08 2019][2937650.677502] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:08 2019][2937650.684483] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:08 2019][2937650.690378] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.697510] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.705265] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:08 2019][2937650.712530] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.720410] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.727405] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:08 2019][2937650.733859] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:08 2019][2937650.739912] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.746399] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.753977] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:08 2019][2937650.759045] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.765318] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:08 2019][2937650.771959] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:08 2019][2937650.778256] mdt02_050 S ffff8b8a7b2f30c0 0 118392 2 0x00000080 [Fri Apr 12 11:12:08 2019][2937650.785631] Call Trace: [Fri Apr 12 11:12:08 2019][2937650.788286] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:08 2019][2937650.795215] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.802971] [] schedule+0x29/0x70 [Fri Apr 12 11:12:08 2019][2937650.808150] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.815236] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.822332] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:08 2019][2937650.829482] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:08 2019][2937650.835272] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.842634] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.850676] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:08 2019][2937650.858453] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:08 2019][2937650.865385] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937650.873169] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937650.881192] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:09 2019][2937650.887858] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:09 2019][2937650.894071] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:09 2019][2937650.900800] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:09 2019][2937650.907786] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:09 2019][2937650.913682] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937650.920793] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937650.928566] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:09 2019][2937650.935832] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937650.943730] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937650.950708] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:09 2019][2937650.957179] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:09 2019][2937650.963241] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937650.969742] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937650.977331] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:09 2019][2937650.982398] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937650.988673] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:09 2019][2937650.995292] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.001568] mdt01_100 S ffff8b8a7b2f4100 0 118393 2 0x00000080 [Fri Apr 12 11:12:09 2019][2937651.008929] Call Trace: [Fri Apr 12 11:12:09 2019][2937651.011569] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:09 2019][2937651.018450] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:09 2019][2937651.024589] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.032334] [] schedule+0x29/0x70 [Fri Apr 12 11:12:09 2019][2937651.037519] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.044605] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.051682] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.058823] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:09 2019][2937651.064611] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.071959] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.080001] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.087766] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.094250] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.102036] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.110054] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.116704] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.124386] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.130672] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:09 2019][2937651.136606] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:09 2019][2937651.143153] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:09 2019][2937651.149350] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.156067] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.157084] LustreError: dumping log to /tmp/lustre-log.1555092729.118484 [Fri Apr 12 11:12:09 2019][2937651.170009] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:09 2019][2937651.175903] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.182997] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.190744] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:09 2019][2937651.198007] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.205876] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.212840] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:09 2019][2937651.219286] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:09 2019][2937651.225342] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.231821] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.239396] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:09 2019][2937651.244457] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.250751] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:09 2019][2937651.257369] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.263645] mdt00_068 S ffff8b8a7b2f5140 0 118394 2 0x00000080 [Fri Apr 12 11:12:09 2019][2937651.271007] Call Trace: [Fri Apr 12 11:12:09 2019][2937651.273645] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:09 2019][2937651.280565] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.288312] [] schedule+0x29/0x70 [Fri Apr 12 11:12:09 2019][2937651.293492] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.300573] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.307654] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.314811] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:09 2019][2937651.320594] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.327943] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.335962] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.342681] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:09 2019][2937651.349655] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.356397] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.364448] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.371515] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.378690] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.386114] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:12:09 2019][2937651.393201] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.399858] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.406943] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.414093] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:09 2019][2937651.420811] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:09 2019][2937651.427441] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.434265] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.441150] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:09 2019][2937651.448470] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:09 2019][2937651.455035] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.462299] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.470004] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.476318] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.483405] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.491156] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:09 2019][2937651.498413] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.506281] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.513252] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:09 2019][2937651.519040] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.525525] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.533097] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:09 2019][2937651.538158] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.544434] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:09 2019][2937651.551057] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.557328] mdt03_035 S ffff8b8a7b2f6180 0 118395 2 0x00000080 [Fri Apr 12 11:12:09 2019][2937651.564684] Call Trace: [Fri Apr 12 11:12:09 2019][2937651.567315] [] schedule+0x29/0x70 [Fri Apr 12 11:12:09 2019][2937651.572499] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.579377] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:09 2019][2937651.585171] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.591651] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.599225] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:09 2019][2937651.604287] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.610563] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:09 2019][2937651.617182] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.623449] mdt00_069 S ffff8b8c5d640000 0 118396 2 0x00000080 [Fri Apr 12 11:12:09 2019][2937651.630812] Call Trace: [Fri Apr 12 11:12:09 2019][2937651.633444] [] schedule+0x29/0x70 [Fri Apr 12 11:12:09 2019][2937651.638631] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.645509] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:09 2019][2937651.651301] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.657783] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.665354] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:09 2019][2937651.670414] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.676691] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:09 2019][2937651.683311] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:09 2019][2937651.689585] mdt02_051 S ffff8b8c5d641040 0 118397 2 0x00000080 [Fri Apr 12 11:12:09 2019][2937651.696948] Call Trace: [Fri Apr 12 11:12:09 2019][2937651.699616] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.707362] [] schedule+0x29/0x70 [Fri Apr 12 11:12:09 2019][2937651.712543] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.719623] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.726673] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:09 2019][2937651.732461] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.739716] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.747064] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.753891] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.761905] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:09 2019][2937651.768532] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:09 2019][2937651.775267] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:09 2019][2937651.781652] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.788707] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:09 2019][2937651.796386] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.803711] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:09 2019][2937651.809999] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:09 2019][2937651.817407] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:09 2019][2937651.824297] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.830696] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:09 2019][2937651.837845] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.844834] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:09 2019][2937651.851476] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.859161] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:09 2019][2937651.865446] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:09 2019][2937651.871386] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:10 2019][2937651.877930] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:10 2019][2937651.884124] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:10 2019][2937651.890842] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:10 2019][2937651.897819] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:10 2019][2937651.903705] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937651.910794] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937651.918548] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:10 2019][2937651.925813] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937651.933682] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937651.940650] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:10 2019][2937651.947095] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:10 2019][2937651.953147] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937651.959630] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937651.967203] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:10 2019][2937651.972265] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:10 2019][2937651.978557] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:10 2019][2937651.985176] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:10 2019][2937651.991444] mdt03_036 S ffff8b8c5d642080 0 118398 2 0x00000080 [Fri Apr 12 11:12:10 2019][2937651.998808] Call Trace: [Fri Apr 12 11:12:10 2019][2937652.001438] [] schedule+0x29/0x70 [Fri Apr 12 11:12:10 2019][2937652.006623] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.013502] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:10 2019][2937652.019293] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.025774] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.033350] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:10 2019][2937652.038423] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:10 2019][2937652.044694] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:10 2019][2937652.051313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:10 2019][2937652.057579] mdt00_070 S ffff8b8c5d6430c0 0 118399 2 0x00000080 [Fri Apr 12 11:12:10 2019][2937652.064935] Call Trace: [Fri Apr 12 11:12:10 2019][2937652.067602] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.075350] [] schedule+0x29/0x70 [Fri Apr 12 11:12:10 2019][2937652.080531] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.087617] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.094667] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:10 2019][2937652.100455] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.107700] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:10 2019][2937652.115054] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.121878] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.129900] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:10 2019][2937652.136527] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:10 2019][2937652.143246] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:10 2019][2937652.149632] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:10 2019][2937652.156694] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:10 2019][2937652.164380] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:10 2019][2937652.171697] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:10 2019][2937652.177982] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:10 2019][2937652.185396] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:10 2019][2937652.192287] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.198684] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.205831] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.212808] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.219454] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:10 2019][2937652.227138] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:10 2019][2937652.233425] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:10 2019][2937652.239377] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:10 2019][2937652.245925] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:10 2019][2937652.252119] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.258839] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.265815] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:10 2019][2937652.271704] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.278802] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.286557] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:10 2019][2937652.293820] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.301695] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.308676] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:10 2019][2937652.315117] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:10 2019][2937652.321170] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.327650] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.335224] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:10 2019][2937652.340284] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:10 2019][2937652.346562] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:10 2019][2937652.353180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:10 2019][2937652.359448] mdt01_101 S ffff8b8c5d644100 0 118400 2 0x00000080 [Fri Apr 12 11:12:10 2019][2937652.366810] Call Trace: [Fri Apr 12 11:12:10 2019][2937652.369448] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:10 2019][2937652.376367] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.384115] [] schedule+0x29/0x70 [Fri Apr 12 11:12:10 2019][2937652.389299] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.396383] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.403457] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:10 2019][2937652.410592] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:10 2019][2937652.416382] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.423734] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.431749] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.438482] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:10 2019][2937652.445459] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.452198] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.460235] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.467329] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.474390] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.481541] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.488171] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:10 2019][2937652.495345] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.502004] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.509094] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.516249] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:10 2019][2937652.522967] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:10 2019][2937652.529597] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.536422] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.543306] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:10 2019][2937652.550628] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:10 2019][2937652.557189] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.564446] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.572150] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.578481] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.585572] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.593332] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:10 2019][2937652.600598] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.608467] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.615435] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:10 2019][2937652.621880] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:10 2019][2937652.627933] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.634413] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.641988] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:10 2019][2937652.647064] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:10 2019][2937652.653334] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:10 2019][2937652.659954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:10 2019][2937652.666228] mdt01_102 S ffff8b8c5d645140 0 118402 2 0x00000080 [Fri Apr 12 11:12:10 2019][2937652.673591] Call Trace: [Fri Apr 12 11:12:10 2019][2937652.676227] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:10 2019][2937652.683143] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.690887] [] schedule+0x29/0x70 [Fri Apr 12 11:12:10 2019][2937652.696071] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.703156] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.710248] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:10 2019][2937652.717390] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:10 2019][2937652.723176] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.730517] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.738561] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.746327] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:10 2019][2937652.753242] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.761016] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.769031] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:10 2019][2937652.775662] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:10 2019][2937652.781871] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.788592] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:10 2019][2937652.795566] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:10 2019][2937652.801451] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.808547] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.816304] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:10 2019][2937652.823569] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.831437] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.838406] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:10 2019][2937652.844853] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:10 2019][2937652.850917] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.857394] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:10 2019][2937652.864969] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:10 2019][2937652.870031] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937652.876305] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:11 2019][2937652.882926] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937652.889201] mdt01_103 S ffff8b8c5d646180 0 118403 2 0x00000080 [Fri Apr 12 11:12:11 2019][2937652.896565] Call Trace: [Fri Apr 12 11:12:11 2019][2937652.899195] [] schedule+0x29/0x70 [Fri Apr 12 11:12:11 2019][2937652.904382] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937652.911258] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:11 2019][2937652.917050] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937652.923532] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937652.931109] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:11 2019][2937652.936166] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937652.942443] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:11 2019][2937652.949064] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937652.955337] mdt03_037 S ffff8b8ccab18000 0 118404 2 0x00000080 [Fri Apr 12 11:12:11 2019][2937652.962694] Call Trace: [Fri Apr 12 11:12:11 2019][2937652.965324] [] schedule+0x29/0x70 [Fri Apr 12 11:12:11 2019][2937652.970511] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937652.977387] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:11 2019][2937652.983177] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937652.989659] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937652.997232] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:11 2019][2937653.002286] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.008561] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:11 2019][2937653.015183] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.021456] mdt02_052 S ffff8b8ccab19040 0 118405 2 0x00000080 [Fri Apr 12 11:12:11 2019][2937653.028812] Call Trace: [Fri Apr 12 11:12:11 2019][2937653.031444] [] schedule+0x29/0x70 [Fri Apr 12 11:12:11 2019][2937653.036630] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.043507] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:11 2019][2937653.049296] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.055780] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.063354] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:11 2019][2937653.068416] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.074692] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:11 2019][2937653.081312] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.087584] mdt01_104 S ffff8b8ccab1a080 0 118406 2 0x00000080 [Fri Apr 12 11:12:11 2019][2937653.094941] Call Trace: [Fri Apr 12 11:12:11 2019][2937653.097608] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.105353] [] schedule+0x29/0x70 [Fri Apr 12 11:12:11 2019][2937653.110535] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.117624] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.124672] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:11 2019][2937653.130459] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.137709] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:11 2019][2937653.145057] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.151883] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.159914] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:11 2019][2937653.166541] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:11 2019][2937653.173258] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:11 2019][2937653.179648] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:11 2019][2937653.186707] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:11 2019][2937653.194393] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:11 2019][2937653.201711] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:11 2019][2937653.207998] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:11 2019][2937653.215408] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:11 2019][2937653.222298] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:11 2019][2937653.228694] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.235844] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:11 2019][2937653.242819] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:11 2019][2937653.249468] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:11 2019][2937653.257154] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:11 2019][2937653.263438] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:11 2019][2937653.269373] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:11 2019][2937653.275920] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:11 2019][2937653.282136] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:11 2019][2937653.288850] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:11 2019][2937653.295827] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:11 2019][2937653.301713] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.308799] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.316554] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:11 2019][2937653.323807] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.331678] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.338650] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:11 2019][2937653.345098] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:11 2019][2937653.351156] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.357640] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.365214] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:11 2019][2937653.370274] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.376550] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:11 2019][2937653.383170] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.389446] mdt00_071 S ffff8b8ccab1b0c0 0 118407 2 0x00000080 [Fri Apr 12 11:12:11 2019][2937653.396808] Call Trace: [Fri Apr 12 11:12:11 2019][2937653.399439] [] schedule+0x29/0x70 [Fri Apr 12 11:12:11 2019][2937653.404625] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.411503] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:11 2019][2937653.417292] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.423788] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.431357] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:11 2019][2937653.436418] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.442693] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:11 2019][2937653.449315] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.455588] mdt02_053 S ffff8b8ccab1c100 0 118408 2 0x00000080 [Fri Apr 12 11:12:11 2019][2937653.462942] Call Trace: [Fri Apr 12 11:12:11 2019][2937653.465573] [] schedule+0x29/0x70 [Fri Apr 12 11:12:11 2019][2937653.470761] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.477640] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:11 2019][2937653.483429] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.489914] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.497488] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:11 2019][2937653.502547] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.508822] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:11 2019][2937653.515442] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.521718] mdt00_072 S ffff8b8ccab1d140 0 118409 2 0x00000080 [Fri Apr 12 11:12:11 2019][2937653.529080] Call Trace: [Fri Apr 12 11:12:11 2019][2937653.531748] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.539495] [] schedule+0x29/0x70 [Fri Apr 12 11:12:11 2019][2937653.544677] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.551765] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.558848] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:11 2019][2937653.565990] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:11 2019][2937653.571797] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.579143] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.587184] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.594943] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:11 2019][2937653.601860] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.609642] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.617665] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:11 2019][2937653.624294] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:11 2019][2937653.630491] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:11 2019][2937653.637207] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:11 2019][2937653.644184] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:11 2019][2937653.650071] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.657166] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.664919] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:11 2019][2937653.672188] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.680054] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.687023] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:11 2019][2937653.693469] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:11 2019][2937653.699524] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.706001] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.713576] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:11 2019][2937653.717133] LustreError: dumping log to /tmp/lustre-log.1555092731.118379 [Fri Apr 12 11:12:11 2019][2937653.725595] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.731864] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:11 2019][2937653.738484] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:11 2019][2937653.744758] mdt01_105 S ffff8b8ccab1e180 0 118410 2 0x00000080 [Fri Apr 12 11:12:11 2019][2937653.752130] Call Trace: [Fri Apr 12 11:12:11 2019][2937653.754796] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.762547] [] schedule+0x29/0x70 [Fri Apr 12 11:12:11 2019][2937653.767726] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.774813] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.781863] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:11 2019][2937653.787653] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.794900] [] ? kfree+0x106/0x140 [Fri Apr 12 11:12:11 2019][2937653.800167] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.806992] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:11 2019][2937653.815025] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:11 2019][2937653.821652] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:11 2019][2937653.828382] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:11 2019][2937653.834771] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:11 2019][2937653.841839] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:11 2019][2937653.849540] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:12 2019][2937653.856867] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:12 2019][2937653.863208] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:12 2019][2937653.870633] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:12 2019][2937653.877524] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:12 2019][2937653.883929] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937653.891077] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:12 2019][2937653.898056] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:12 2019][2937653.904709] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:12 2019][2937653.912397] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:12 2019][2937653.918681] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:12 2019][2937653.924619] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:12 2019][2937653.931163] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:12 2019][2937653.937356] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:12 2019][2937653.944077] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:12 2019][2937653.951068] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:12 2019][2937653.956976] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937653.964082] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937653.971885] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:12 2019][2937653.979151] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937653.987027] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937653.994034] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.000524] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.008096] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:12 2019][2937654.013154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.019450] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:12 2019][2937654.026067] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.032334] mdt_io00_003 S ffff8b8bc1248000 0 118411 2 0x00000080 [Fri Apr 12 11:12:12 2019][2937654.039698] Call Trace: [Fri Apr 12 11:12:12 2019][2937654.042328] [] schedule+0x29/0x70 [Fri Apr 12 11:12:12 2019][2937654.047515] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.054395] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:12 2019][2937654.060186] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.066665] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.074240] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:12 2019][2937654.079317] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.085583] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:12 2019][2937654.092203] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.098471] mdt_io01_003 S ffff8b8bc1249040 0 118412 2 0x00000080 [Fri Apr 12 11:12:12 2019][2937654.105835] Call Trace: [Fri Apr 12 11:12:12 2019][2937654.108466] [] schedule+0x29/0x70 [Fri Apr 12 11:12:12 2019][2937654.113651] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.120528] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:12 2019][2937654.126318] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.132800] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.140390] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:12 2019][2937654.145446] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.151722] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:12 2019][2937654.158342] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.164614] mdt_io02_003 S ffff8b8bc124a080 0 118413 2 0x00000080 [Fri Apr 12 11:12:12 2019][2937654.171970] Call Trace: [Fri Apr 12 11:12:12 2019][2937654.174603] [] schedule+0x29/0x70 [Fri Apr 12 11:12:12 2019][2937654.179789] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.186667] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:12 2019][2937654.192456] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.198937] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.206513] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:12 2019][2937654.211572] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.217840] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:12 2019][2937654.224462] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.229144] LustreError: dumping log to /tmp/lustre-log.1555092732.118393 [Fri Apr 12 11:12:12 2019][2937654.237694] mdt00_073 S ffff8b8bc124b0c0 0 118414 2 0x00000080 [Fri Apr 12 11:12:12 2019][2937654.245050] Call Trace: [Fri Apr 12 11:12:12 2019][2937654.247689] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:12 2019][2937654.254571] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:12 2019][2937654.260707] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.268457] [] schedule+0x29/0x70 [Fri Apr 12 11:12:12 2019][2937654.273636] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.280716] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.287801] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:12 2019][2937654.294941] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:12 2019][2937654.300729] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.308078] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.316123] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.323885] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.330373] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.338154] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.346191] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.352840] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:12 2019][2937654.360522] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:12 2019][2937654.366800] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:12 2019][2937654.372738] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:12 2019][2937654.379284] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:12 2019][2937654.385479] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.392196] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.399171] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:12 2019][2937654.405064] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.412161] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.419914] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:12 2019][2937654.427177] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.435042] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.442010] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:12 2019][2937654.448457] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:12 2019][2937654.454510] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.460991] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.468581] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:12 2019][2937654.473641] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.479912] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:12 2019][2937654.486530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.492803] mdt03_038 S ffff8b8bc124c100 0 118415 2 0x00000080 [Fri Apr 12 11:12:12 2019][2937654.500160] Call Trace: [Fri Apr 12 11:12:12 2019][2937654.502806] [] schedule+0x29/0x70 [Fri Apr 12 11:12:12 2019][2937654.507987] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.514862] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:12 2019][2937654.520654] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.527142] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.534721] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:12 2019][2937654.539799] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.546094] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:12 2019][2937654.552720] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.558995] mdt_io00_004 S ffff8b8bc124d140 0 118416 2 0x00000080 [Fri Apr 12 11:12:12 2019][2937654.566377] Call Trace: [Fri Apr 12 11:12:12 2019][2937654.569008] [] schedule+0x29/0x70 [Fri Apr 12 11:12:12 2019][2937654.574200] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.582242] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.589295] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:12 2019][2937654.595088] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.601758] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.609333] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:12 2019][2937654.614568] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.620854] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:12 2019][2937654.627644] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:12 2019][2937654.633930] mdt02_054 S ffff8b8bc124e180 0 118417 2 0x00000080 [Fri Apr 12 11:12:12 2019][2937654.641292] Call Trace: [Fri Apr 12 11:12:12 2019][2937654.643931] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:12 2019][2937654.650851] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.658598] [] schedule+0x29/0x70 [Fri Apr 12 11:12:12 2019][2937654.663779] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.670881] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.677970] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:12 2019][2937654.685119] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:12 2019][2937654.690954] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.698295] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.706317] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.713035] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:12 2019][2937654.720030] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.726768] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.734815] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.741908] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.748976] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.756125] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.762756] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:12 2019][2937654.769939] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.776598] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.783697] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.790887] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:12 2019][2937654.797632] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:12 2019][2937654.804272] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:12 2019][2937654.811131] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.818058] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:12 2019][2937654.825403] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:12 2019][2937654.831991] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.839270] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.846981] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:12 2019][2937654.853293] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.860397] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.868152] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:13 2019][2937654.875465] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.883375] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.890359] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:13 2019][2937654.896813] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:13 2019][2937654.902867] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.909346] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.916921] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:13 2019][2937654.921979] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937654.928258] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:13 2019][2937654.934878] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937654.941150] mdt_io03_003 S ffff8b8ccfbe0000 0 118418 2 0x00000080 [Fri Apr 12 11:12:13 2019][2937654.948516] Call Trace: [Fri Apr 12 11:12:13 2019][2937654.951148] [] schedule+0x29/0x70 [Fri Apr 12 11:12:13 2019][2937654.956332] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.963208] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:13 2019][2937654.968998] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.975484] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937654.983056] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:13 2019][2937654.988118] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937654.994394] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:13 2019][2937655.001026] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.007333] mdt00_074 S ffff8b8ccfbe1040 0 118419 2 0x00000080 [Fri Apr 12 11:12:13 2019][2937655.014712] Call Trace: [Fri Apr 12 11:12:13 2019][2937655.017343] [] schedule+0x29/0x70 [Fri Apr 12 11:12:13 2019][2937655.022530] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.029406] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:13 2019][2937655.035195] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.041679] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.049253] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:13 2019][2937655.054313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.060599] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:13 2019][2937655.067230] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.073512] mdt01_106 S ffff8b8ccfbe2080 0 118420 2 0x00000080 [Fri Apr 12 11:12:13 2019][2937655.080893] Call Trace: [Fri Apr 12 11:12:13 2019][2937655.083527] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:13 2019][2937655.089334] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.097106] [] schedule+0x29/0x70 [Fri Apr 12 11:12:13 2019][2937655.102288] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.109372] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.116470] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:13 2019][2937655.123637] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:13 2019][2937655.129444] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.136787] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.144831] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.152610] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:13 2019][2937655.159545] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.167378] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.175408] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:13 2019][2937655.182055] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:13 2019][2937655.188256] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:13 2019][2937655.194980] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:13 2019][2937655.201979] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:13 2019][2937655.207891] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.214991] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.222774] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:13 2019][2937655.230047] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.237940] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.244930] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:13 2019][2937655.251384] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:13 2019][2937655.257439] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.263972] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.271557] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:13 2019][2937655.276638] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.282914] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:13 2019][2937655.289538] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.295808] mdt00_075 S ffff8b8ccfbe30c0 0 118421 2 0x00000080 [Fri Apr 12 11:12:13 2019][2937655.303181] Call Trace: [Fri Apr 12 11:12:13 2019][2937655.305815] [] schedule+0x29/0x70 [Fri Apr 12 11:12:13 2019][2937655.310979] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:12:13 2019][2937655.317022] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:12:13 2019][2937655.323576] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Fri Apr 12 11:12:13 2019][2937655.330503] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.338308] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.345399] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.352506] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:13 2019][2937655.358302] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.365564] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:13 2019][2937655.372931] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.379800] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.387823] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:13 2019][2937655.394462] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:13 2019][2937655.401185] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:13 2019][2937655.407594] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:13 2019][2937655.414662] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:13 2019][2937655.422374] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:13 2019][2937655.429699] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:13 2019][2937655.435992] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:13 2019][2937655.443400] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:13 2019][2937655.450312] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:13 2019][2937655.456723] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.463881] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:13 2019][2937655.470869] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:13 2019][2937655.477527] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:13 2019][2937655.485224] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:13 2019][2937655.491538] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:13 2019][2937655.497489] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:13 2019][2937655.504058] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:13 2019][2937655.510361] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:13 2019][2937655.517182] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:13 2019][2937655.524178] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:13 2019][2937655.530089] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.537216] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.544982] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:13 2019][2937655.552268] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.560173] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.567145] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:13 2019][2937655.573609] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:13 2019][2937655.579701] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.586201] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.593778] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:13 2019][2937655.598835] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.605112] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:13 2019][2937655.611732] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.618046] mdt03_039 S ffff8b8ccfbe4100 0 118422 2 0x00000080 [Fri Apr 12 11:12:13 2019][2937655.625451] Call Trace: [Fri Apr 12 11:12:13 2019][2937655.628092] [] schedule+0x29/0x70 [Fri Apr 12 11:12:13 2019][2937655.633290] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.640180] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:13 2019][2937655.645971] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.652496] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.660106] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:13 2019][2937655.665168] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.671440] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:13 2019][2937655.678071] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.684371] mdt01_107 S ffff8b8ccfbe5140 0 118423 2 0x00000080 [Fri Apr 12 11:12:13 2019][2937655.691759] Call Trace: [Fri Apr 12 11:12:13 2019][2937655.694394] [] schedule+0x29/0x70 [Fri Apr 12 11:12:13 2019][2937655.699644] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.706534] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:13 2019][2937655.712326] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.718830] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.726408] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:13 2019][2937655.731469] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.737761] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:13 2019][2937655.744390] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:13 2019][2937655.750660] mdt03_040 S ffff8b8ccfbe6180 0 118424 2 0x00000080 [Fri Apr 12 11:12:13 2019][2937655.758016] Call Trace: [Fri Apr 12 11:12:13 2019][2937655.760648] [] schedule+0x29/0x70 [Fri Apr 12 11:12:13 2019][2937655.765839] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.772721] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:13 2019][2937655.778511] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:13 2019][2937655.785023] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.792642] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:14 2019][2937655.797709] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937655.803988] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:14 2019][2937655.810613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937655.816884] mdt_io00_005 S ffff8b8b083a0000 0 118425 2 0x00000080 [Fri Apr 12 11:12:14 2019][2937655.824250] Call Trace: [Fri Apr 12 11:12:14 2019][2937655.826907] [] schedule+0x29/0x70 [Fri Apr 12 11:12:14 2019][2937655.832099] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.838999] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:14 2019][2937655.844795] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.851288] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.858873] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:14 2019][2937655.863933] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937655.870243] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:14 2019][2937655.877126] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937655.883415] mdt00_076 R running task 0 118426 2 0x00000080 [Fri Apr 12 11:12:14 2019][2937655.890812] Call Trace: [Fri Apr 12 11:12:14 2019][2937655.893446] [] schedule+0x29/0x70 [Fri Apr 12 11:12:14 2019][2937655.898656] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.905544] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:14 2019][2937655.911349] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.917846] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.925455] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:14 2019][2937655.930532] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937655.936816] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:14 2019][2937655.943432] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937655.949713] mdt03_041 S ffff8b8b083a2080 0 118427 2 0x00000080 [Fri Apr 12 11:12:14 2019][2937655.957097] Call Trace: [Fri Apr 12 11:12:14 2019][2937655.959727] [] schedule+0x29/0x70 [Fri Apr 12 11:12:14 2019][2937655.964922] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.971799] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:14 2019][2937655.977595] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.984085] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937655.991694] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:14 2019][2937655.996751] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937656.003046] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:14 2019][2937656.009670] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937656.015957] mdt01_108 S ffff8b8b083a30c0 0 118428 2 0x00000080 [Fri Apr 12 11:12:14 2019][2937656.023398] Call Trace: [Fri Apr 12 11:12:14 2019][2937656.026039] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:14 2019][2937656.032959] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.040719] [] schedule+0x29/0x70 [Fri Apr 12 11:12:14 2019][2937656.045950] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.054294] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.061452] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.068640] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:14 2019][2937656.074464] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.081856] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.089936] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.097769] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:14 2019][2937656.104701] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.112486] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.120531] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:14 2019][2937656.127168] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:14 2019][2937656.133390] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.140135] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.147112] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:14 2019][2937656.153007] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.160123] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.167928] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:14 2019][2937656.175250] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.183200] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.190172] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:14 2019][2937656.196652] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:14 2019][2937656.202724] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.209244] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.216838] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:14 2019][2937656.221910] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937656.228200] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:14 2019][2937656.234832] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937656.241102] mdt_io01_004 S ffff8b8b083a4100 0 118429 2 0x00000080 [Fri Apr 12 11:12:14 2019][2937656.248466] Call Trace: [Fri Apr 12 11:12:14 2019][2937656.251098] [] schedule+0x29/0x70 [Fri Apr 12 11:12:14 2019][2937656.256288] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.263173] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:14 2019][2937656.268966] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.275464] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.283046] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:14 2019][2937656.288124] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937656.294398] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:14 2019][2937656.301018] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937656.307290] mdt00_077 S ffff8b8b083a5140 0 118430 2 0x00000080 [Fri Apr 12 11:12:14 2019][2937656.314680] Call Trace: [Fri Apr 12 11:12:14 2019][2937656.317350] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.325093] [] schedule+0x29/0x70 [Fri Apr 12 11:12:14 2019][2937656.330278] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.337382] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.344453] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:14 2019][2937656.350270] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.357538] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.364911] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.371742] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.379773] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:14 2019][2937656.386405] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:14 2019][2937656.393148] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:14 2019][2937656.399539] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.406596] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:14 2019][2937656.414281] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.421599] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:14 2019][2937656.427885] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:14 2019][2937656.435294] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:14 2019][2937656.442193] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.448617] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.455769] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.462760] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.469413] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.477101] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.483417] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:14 2019][2937656.489376] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:14 2019][2937656.495942] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:14 2019][2937656.502155] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.508872] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.515854] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:14 2019][2937656.521754] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.528882] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.536655] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:14 2019][2937656.543926] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.551799] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.558784] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:14 2019][2937656.565240] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:14 2019][2937656.571312] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.577808] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.585441] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:14 2019][2937656.590521] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937656.596811] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:14 2019][2937656.603442] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:14 2019][2937656.609715] mdt02_055 S ffff8b8b083a6180 0 118431 2 0x00000080 [Fri Apr 12 11:12:14 2019][2937656.617095] Call Trace: [Fri Apr 12 11:12:14 2019][2937656.619786] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.627537] [] schedule+0x29/0x70 [Fri Apr 12 11:12:14 2019][2937656.632716] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.639831] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.646909] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:14 2019][2937656.652702] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.659957] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.667305] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.674135] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.682155] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:14 2019][2937656.688781] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:14 2019][2937656.695500] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:14 2019][2937656.701890] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.708948] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:14 2019][2937656.716660] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.723987] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:14 2019][2937656.730273] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:14 2019][2937656.737681] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:14 2019][2937656.744575] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.750976] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:14 2019][2937656.758130] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.765105] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.771756] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.779437] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:14 2019][2937656.785734] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:14 2019][2937656.791669] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:14 2019][2937656.798217] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:14 2019][2937656.804434] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.811175] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:14 2019][2937656.818165] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:14 2019][2937656.824060] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.831185] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.838963] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:15 2019][2937656.846266] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.854171] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.861153] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:15 2019][2937656.867723] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:15 2019][2937656.873777] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.880292] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.887882] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:15 2019][2937656.892940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937656.899254] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:15 2019][2937656.905883] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937656.912186] mdt_io01_005 S ffff8b8b23f08000 0 118432 2 0x00000080 [Fri Apr 12 11:12:15 2019][2937656.919559] Call Trace: [Fri Apr 12 11:12:15 2019][2937656.922196] [] schedule+0x29/0x70 [Fri Apr 12 11:12:15 2019][2937656.927427] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.934311] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:15 2019][2937656.940110] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.946601] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937656.954180] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:15 2019][2937656.959252] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937656.965526] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:15 2019][2937656.972165] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937656.978437] mdt_io02_004 S ffff8b8b23f09040 0 118433 2 0x00000080 [Fri Apr 12 11:12:15 2019][2937656.985850] Call Trace: [Fri Apr 12 11:12:15 2019][2937656.988482] [] schedule+0x29/0x70 [Fri Apr 12 11:12:15 2019][2937656.993678] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.000555] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:15 2019][2937657.006349] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.012830] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.020403] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:15 2019][2937657.025464] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937657.031746] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:15 2019][2937657.038399] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937657.044670] mdt03_042 S ffff8b8b23f0a080 0 118434 2 0x00000080 [Fri Apr 12 11:12:15 2019][2937657.052058] Call Trace: [Fri Apr 12 11:12:15 2019][2937657.054730] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.062483] [] schedule+0x29/0x70 [Fri Apr 12 11:12:15 2019][2937657.067690] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.074781] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.081871] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:15 2019][2937657.087662] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.094928] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:15 2019][2937657.102285] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.109136] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.117209] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:15 2019][2937657.123842] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:15 2019][2937657.130564] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:15 2019][2937657.136950] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:15 2019][2937657.144038] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:15 2019][2937657.151731] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:15 2019][2937657.159084] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:15 2019][2937657.165372] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:15 2019][2937657.172799] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:15 2019][2937657.179705] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.186120] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.193276] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.200251] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.206935] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:15 2019][2937657.214630] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:15 2019][2937657.220916] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:15 2019][2937657.226848] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:15 2019][2937657.233408] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:15 2019][2937657.239619] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.246344] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.253331] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:15 2019][2937657.259316] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.266410] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.274184] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:15 2019][2937657.281472] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.289349] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.296314] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:15 2019][2937657.302107] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.308598] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.316181] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:15 2019][2937657.321236] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937657.327512] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:15 2019][2937657.334151] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937657.340424] mdt00_078 S ffff8b8b23f0b0c0 0 118435 2 0x00000080 [Fri Apr 12 11:12:15 2019][2937657.347800] Call Trace: [Fri Apr 12 11:12:15 2019][2937657.350449] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:15 2019][2937657.357389] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.365159] [] schedule+0x29/0x70 [Fri Apr 12 11:12:15 2019][2937657.370348] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.377431] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.384516] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:15 2019][2937657.391667] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:15 2019][2937657.397468] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.404824] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.412926] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.419689] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:15 2019][2937657.426675] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.433425] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.441487] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.448651] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.455725] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.462885] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.469513] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:15 2019][2937657.476695] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.483360] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.490452] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.497602] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:15 2019][2937657.504329] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:15 2019][2937657.511020] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.517937] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.524773] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.531671] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:15 2019][2937657.538993] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:15 2019][2937657.545665] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.552981] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.560821] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.567224] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.574420] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.582282] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:15 2019][2937657.589623] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.597638] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.604735] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:15 2019][2937657.611261] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:15 2019][2937657.617412] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.623951] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.631562] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:15 2019][2937657.636628] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937657.642901] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:15 2019][2937657.649533] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937657.655822] mdt01_109 S ffff8b8b23f0c100 0 118437 2 0x00000080 [Fri Apr 12 11:12:15 2019][2937657.663185] Call Trace: [Fri Apr 12 11:12:15 2019][2937657.665828] [] schedule+0x29/0x70 [Fri Apr 12 11:12:15 2019][2937657.671026] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.677936] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:15 2019][2937657.683731] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.690219] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.697813] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:15 2019][2937657.702895] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937657.709170] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:15 2019][2937657.715827] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:15 2019][2937657.722097] mdt02_056 S ffff8b8b23f0d140 0 118438 2 0x00000080 [Fri Apr 12 11:12:15 2019][2937657.729478] Call Trace: [Fri Apr 12 11:12:15 2019][2937657.732119] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:15 2019][2937657.739035] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.746785] [] schedule+0x29/0x70 [Fri Apr 12 11:12:15 2019][2937657.751966] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.759051] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.766138] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:15 2019][2937657.773277] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:15 2019][2937657.779065] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.786412] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.794470] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.802252] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:15 2019][2937657.809176] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.816984] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:15 2019][2937657.825037] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:15 2019][2937657.831680] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:15 2019][2937657.837881] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.844644] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:15 2019][2937657.851626] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:15 2019][2937657.857549] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.864645] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.872402] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:16 2019][2937657.879678] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.887590] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.894582] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:16 2019][2937657.901037] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:16 2019][2937657.907104] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.913588] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.921159] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937657.926230] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937657.932505] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937657.939127] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937657.945399] mdt00_079 S ffff8b8b23f0e180 0 118439 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937657.952783] Call Trace: [Fri Apr 12 11:12:16 2019][2937657.955431] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937657.960633] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.967521] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:16 2019][2937657.973340] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.979842] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937657.987419] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937657.992478] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937657.998772] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937658.005390] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.011667] mdt00_080 S ffff8b8a95618000 0 118440 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937658.019038] Call Trace: [Fri Apr 12 11:12:16 2019][2937658.021680] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:16 2019][2937658.028603] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.036381] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937658.041562] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.048672] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.055760] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:16 2019][2937658.062961] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:16 2019][2937658.068768] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.076117] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.084168] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.091945] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:16 2019][2937658.098887] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.106686] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.114731] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:16 2019][2937658.121370] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:16 2019][2937658.127571] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:16 2019][2937658.134290] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:16 2019][2937658.141263] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:16 2019][2937658.147159] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.154257] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.162009] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:16 2019][2937658.169276] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.177145] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.184113] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:16 2019][2937658.190558] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:16 2019][2937658.196614] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.203097] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.210666] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937658.215729] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.222002] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937658.228623] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.234898] mdt00_081 S ffff8b8a95619040 0 118441 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937658.242262] Call Trace: [Fri Apr 12 11:12:16 2019][2937658.244892] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937658.250081] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.256965] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:16 2019][2937658.262758] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.269241] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.276819] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937658.281880] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.288157] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937658.294778] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.301050] mdt02_057 S ffff8b8a9561a080 0 118442 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937658.308408] Call Trace: [Fri Apr 12 11:12:16 2019][2937658.311037] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937658.316228] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.323107] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:16 2019][2937658.328896] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.335378] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.342949] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937658.348008] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.354286] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937658.360905] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.367179] mdt_io00_006 S ffff8b8a9561b0c0 0 118443 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937658.374544] Call Trace: [Fri Apr 12 11:12:16 2019][2937658.377175] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937658.382364] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.389248] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:16 2019][2937658.395040] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.401523] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.409094] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937658.414157] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.420431] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937658.427050] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.433323] mdt_io02_005 S ffff8b8a9561c100 0 118444 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937658.440680] Call Trace: [Fri Apr 12 11:12:16 2019][2937658.443311] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937658.448499] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.455376] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:16 2019][2937658.461167] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.467648] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.475224] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937658.480284] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.486559] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937658.493180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.499451] mdt_io01_006 S ffff8b8a9561d140 0 118445 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937658.506807] Call Trace: [Fri Apr 12 11:12:16 2019][2937658.509440] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937658.514625] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.521505] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:16 2019][2937658.527294] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.533778] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.541351] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937658.546411] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.552686] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937658.559307] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.565597] mdt02_058 S ffff8b8a9561e180 0 118447 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937658.572953] Call Trace: [Fri Apr 12 11:12:16 2019][2937658.575592] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:16 2019][2937658.582476] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:16 2019][2937658.588613] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.596358] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937658.601541] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.608632] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.615713] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:16 2019][2937658.622852] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:16 2019][2937658.628641] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.635991] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.644032] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.651789] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:16 2019][2937658.658262] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.666038] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.674051] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:16 2019][2937658.680699] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:16 2019][2937658.688383] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:16 2019][2937658.694669] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:16 2019][2937658.700606] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:16 2019][2937658.707152] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:16 2019][2937658.713347] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:16 2019][2937658.720063] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:16 2019][2937658.727042] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:16 2019][2937658.732931] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.740031] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.747783] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:16 2019][2937658.755047] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.762923] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.769890] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:16 2019][2937658.776334] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:16 2019][2937658.782391] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.788871] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.796443] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:16 2019][2937658.801502] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.807777] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:16 2019][2937658.814399] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:16 2019][2937658.820674] mdt01_110 S ffff8b8b86c40000 0 118448 2 0x00000080 [Fri Apr 12 11:12:16 2019][2937658.828036] Call Trace: [Fri Apr 12 11:12:16 2019][2937658.830674] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:16 2019][2937658.837594] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.845342] [] schedule+0x29/0x70 [Fri Apr 12 11:12:16 2019][2937658.850524] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.857611] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:16 2019][2937658.864693] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:17 2019][2937658.871838] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:17 2019][2937658.877629] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.884973] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.893018] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.900789] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:17 2019][2937658.907709] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.915487] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.923500] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:17 2019][2937658.930133] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:17 2019][2937658.936327] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:17 2019][2937658.943046] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:17 2019][2937658.950022] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:17 2019][2937658.955911] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.963010] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.970764] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:17 2019][2937658.978026] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.985896] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937658.992863] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:17 2019][2937658.999310] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:17 2019][2937659.005362] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.011847] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.019424] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:17 2019][2937659.024486] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.030761] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:17 2019][2937659.037398] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.043672] mdt00_082 S ffff8b8b86c41040 0 118449 2 0x00000080 [Fri Apr 12 11:12:17 2019][2937659.051034] Call Trace: [Fri Apr 12 11:12:17 2019][2937659.053666] [] schedule+0x29/0x70 [Fri Apr 12 11:12:17 2019][2937659.058855] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.065731] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:17 2019][2937659.071524] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.078006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.085578] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:17 2019][2937659.090637] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.096933] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:17 2019][2937659.103552] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.109824] mdt02_059 S ffff8b8b86c42080 0 118450 2 0x00000080 [Fri Apr 12 11:12:17 2019][2937659.117180] Call Trace: [Fri Apr 12 11:12:17 2019][2937659.119812] [] schedule+0x29/0x70 [Fri Apr 12 11:12:17 2019][2937659.124999] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.131878] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:17 2019][2937659.137666] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.144149] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.151721] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:17 2019][2937659.156784] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.163057] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:17 2019][2937659.169680] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.175954] mdt00_083 S ffff8b8b86c430c0 0 118451 2 0x00000080 [Fri Apr 12 11:12:17 2019][2937659.183317] Call Trace: [Fri Apr 12 11:12:17 2019][2937659.185955] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:17 2019][2937659.192874] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.200622] [] schedule+0x29/0x70 [Fri Apr 12 11:12:17 2019][2937659.205804] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.212891] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.219965] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:17 2019][2937659.227108] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:17 2019][2937659.232899] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.240248] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.248287] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.256047] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:17 2019][2937659.262961] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.270744] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.278766] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:17 2019][2937659.285396] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:17 2019][2937659.291606] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:17 2019][2937659.298326] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:17 2019][2937659.305302] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:17 2019][2937659.311191] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.318279] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.326037] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:17 2019][2937659.333300] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.341165] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.348133] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:17 2019][2937659.354578] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:17 2019][2937659.360635] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.367113] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.374688] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:17 2019][2937659.379747] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.386025] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:17 2019][2937659.392644] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.398917] mdt03_043 S ffff8b8b86c44100 0 118452 2 0x00000080 [Fri Apr 12 11:12:17 2019][2937659.406274] Call Trace: [Fri Apr 12 11:12:17 2019][2937659.408904] [] schedule+0x29/0x70 [Fri Apr 12 11:12:17 2019][2937659.414093] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.420977] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:17 2019][2937659.426769] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.433253] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.440831] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:17 2019][2937659.445893] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.452169] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:17 2019][2937659.458788] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.465063] ldlm_cn01_007 S ffff8b8b86c45140 0 118453 2 0x00000080 [Fri Apr 12 11:12:17 2019][2937659.472426] Call Trace: [Fri Apr 12 11:12:17 2019][2937659.475057] [] schedule+0x29/0x70 [Fri Apr 12 11:12:17 2019][2937659.480247] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.487128] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:17 2019][2937659.492916] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.499395] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.506969] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:17 2019][2937659.512031] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:12:17 2019][2937659.518304] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.524582] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:17 2019][2937659.531201] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.537473] mdt03_044 S ffff8b8b86c46180 0 118454 2 0x00000080 [Fri Apr 12 11:12:17 2019][2937659.544847] Call Trace: [Fri Apr 12 11:12:17 2019][2937659.547484] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:17 2019][2937659.554406] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.562159] [] schedule+0x29/0x70 [Fri Apr 12 11:12:17 2019][2937659.567351] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.574437] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.581539] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:17 2019][2937659.588725] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:17 2019][2937659.594540] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.601890] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.609907] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:17 2019][2937659.616649] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:17 2019][2937659.623628] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:17 2019][2937659.630377] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.638458] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.645547] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:17 2019][2937659.652723] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.660131] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:12:17 2019][2937659.667259] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.673921] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.681012] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.688160] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:17 2019][2937659.694897] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:17 2019][2937659.701550] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:17 2019][2937659.708375] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.715266] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:17 2019][2937659.722589] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:17 2019][2937659.729152] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.736434] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.744173] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.750488] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.757586] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.765360] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:17 2019][2937659.772673] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.780614] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.787577] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:17 2019][2937659.794051] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:17 2019][2937659.800105] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.806584] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.814156] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:17 2019][2937659.819217] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.825484] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:17 2019][2937659.832104] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:17 2019][2937659.838379] mdt00_084 S ffff8b8c6b7a0000 0 118455 2 0x00000080 [Fri Apr 12 11:12:17 2019][2937659.845735] Call Trace: [Fri Apr 12 11:12:17 2019][2937659.848373] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:17 2019][2937659.855292] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:17 2019][2937659.863040] [] schedule+0x29/0x70 [Fri Apr 12 11:12:17 2019][2937659.868222] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.875311] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.882406] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:18 2019][2937659.889544] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:18 2019][2937659.895331] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.902689] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.910709] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:18 2019][2937659.917426] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:18 2019][2937659.924403] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:18 2019][2937659.931144] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.939191] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.946283] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.953349] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:18 2019][2937659.960500] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:18 2019][2937659.967132] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:18 2019][2937659.974308] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.980964] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.988052] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937659.995203] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:18 2019][2937660.001920] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:18 2019][2937660.008571] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.015465] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.022308] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.029197] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.036523] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.043094] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.050361] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.058063] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.064376] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.071465] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.079222] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.086483] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.094355] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.101327] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:18 2019][2937660.107773] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:18 2019][2937660.113826] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.120307] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.127879] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:18 2019][2937660.132940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:18 2019][2937660.139209] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:18 2019][2937660.145828] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:18 2019][2937660.152102] mdt_io02_006 S ffff8b8c6b7a1040 0 118456 2 0x00000080 [Fri Apr 12 11:12:18 2019][2937660.159458] Call Trace: [Fri Apr 12 11:12:18 2019][2937660.162089] [] schedule+0x29/0x70 [Fri Apr 12 11:12:18 2019][2937660.167278] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.174159] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:18 2019][2937660.179945] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.186427] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.194001] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:18 2019][2937660.199060] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:18 2019][2937660.205336] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:18 2019][2937660.211957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:18 2019][2937660.218230] mdt03_045 S ffff8b8c6b7a2080 0 118457 2 0x00000080 [Fri Apr 12 11:12:18 2019][2937660.225585] Call Trace: [Fri Apr 12 11:12:18 2019][2937660.228226] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.235141] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.242888] [] schedule+0x29/0x70 [Fri Apr 12 11:12:18 2019][2937660.248072] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.255159] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.262235] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:18 2019][2937660.269377] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:18 2019][2937660.275164] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.282514] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.290532] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.297251] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:18 2019][2937660.304227] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.310967] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.319014] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.326106] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.333167] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.340317] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.346964] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:18 2019][2937660.354142] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.360798] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.367889] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.375037] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:18 2019][2937660.381754] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:18 2019][2937660.388405] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.395299] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.402124] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.409015] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.416336] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.422909] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.430174] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.437879] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.444194] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.451291] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.459046] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.466306] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.474175] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.481141] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:18 2019][2937660.487606] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:18 2019][2937660.493660] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.500142] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.507714] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:18 2019][2937660.512775] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:18 2019][2937660.519052] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:18 2019][2937660.525672] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:18 2019][2937660.531943] mdt01_111 S ffff8b8c6b7a30c0 0 118458 2 0x00000080 [Fri Apr 12 11:12:18 2019][2937660.539300] Call Trace: [Fri Apr 12 11:12:18 2019][2937660.541968] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.549714] [] schedule+0x29/0x70 [Fri Apr 12 11:12:18 2019][2937660.554896] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.561982] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.569034] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:18 2019][2937660.574820] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.582080] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:18 2019][2937660.589424] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.596255] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.604275] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:18 2019][2937660.610899] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:18 2019][2937660.617611] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:18 2019][2937660.623999] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:18 2019][2937660.631059] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.638739] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:18 2019][2937660.646063] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:18 2019][2937660.652350] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:18 2019][2937660.659780] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:18 2019][2937660.666668] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.673070] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.680223] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.687200] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.693849] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:18 2019][2937660.701531] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:18 2019][2937660.707815] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:18 2019][2937660.713753] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:18 2019][2937660.720300] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:18 2019][2937660.726494] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.733211] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:18 2019][2937660.740188] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:18 2019][2937660.746077] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.753165] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.760917] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:18 2019][2937660.768182] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.776053] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.783019] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:18 2019][2937660.789465] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:18 2019][2937660.795520] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.801999] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.809572] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:18 2019][2937660.814634] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:18 2019][2937660.820911] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:18 2019][2937660.827529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:18 2019][2937660.833805] mdt00_085 R running task 0 118459 2 0x00000080 [Fri Apr 12 11:12:18 2019][2937660.841169] Call Trace: [Fri Apr 12 11:12:18 2019][2937660.843800] [] schedule+0x29/0x70 [Fri Apr 12 11:12:18 2019][2937660.848989] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.855874] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:18 2019][2937660.861667] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:18 2019][2937660.868144] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937660.875719] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:19 2019][2937660.880779] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937660.887053] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:19 2019][2937660.893674] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937660.899949] mdt02_060 S ffff8b8c6b7a5140 0 118460 2 0x00000080 [Fri Apr 12 11:12:19 2019][2937660.907313] Call Trace: [Fri Apr 12 11:12:19 2019][2937660.909946] [] schedule+0x29/0x70 [Fri Apr 12 11:12:19 2019][2937660.915134] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937660.922017] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:19 2019][2937660.927810] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937660.934296] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937660.941872] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:19 2019][2937660.946934] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937660.953209] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:19 2019][2937660.959830] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937660.966103] mdt03_046 S ffff8b8c6b7a6180 0 118461 2 0x00000080 [Fri Apr 12 11:12:19 2019][2937660.973458] Call Trace: [Fri Apr 12 11:12:19 2019][2937660.976088] [] schedule+0x29/0x70 [Fri Apr 12 11:12:19 2019][2937660.981284] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937660.988171] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:19 2019][2937660.993964] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.000445] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.008016] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:19 2019][2937661.013078] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.019353] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:19 2019][2937661.025973] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.032238] mdt02_061 S ffff8b8bddb70000 0 118462 2 0x00000080 [Fri Apr 12 11:12:19 2019][2937661.039594] Call Trace: [Fri Apr 12 11:12:19 2019][2937661.042226] [] schedule+0x29/0x70 [Fri Apr 12 11:12:19 2019][2937661.047417] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.054297] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:19 2019][2937661.060107] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.066593] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.074162] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:19 2019][2937661.079224] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.085500] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:19 2019][2937661.092121] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.098394] mdt03_047 S ffff8b8bddb71040 0 118463 2 0x00000080 [Fri Apr 12 11:12:19 2019][2937661.105759] Call Trace: [Fri Apr 12 11:12:19 2019][2937661.108394] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:19 2019][2937661.115312] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.123064] [] schedule+0x29/0x70 [Fri Apr 12 11:12:19 2019][2937661.128243] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.135331] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.142404] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:19 2019][2937661.149549] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:19 2019][2937661.155336] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.162686] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.170706] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.177425] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:19 2019][2937661.184399] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.191154] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.199196] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.206288] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.213355] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.220507] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.227137] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:19 2019][2937661.234316] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.240977] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.248069] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.255217] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:19 2019][2937661.261933] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:19 2019][2937661.268563] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.275387] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.282274] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:19 2019][2937661.289591] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:19 2019][2937661.296157] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.303422] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.311126] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.317440] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.324527] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.332277] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:19 2019][2937661.339544] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.347418] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.354415] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:19 2019][2937661.360865] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:19 2019][2937661.366946] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.373435] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.381097] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:19 2019][2937661.386191] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.392467] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:19 2019][2937661.397290] LustreError: dumping log to /tmp/lustre-log.1555092739.118458 [Fri Apr 12 11:12:19 2019][2937661.406062] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.412336] mdt02_062 S ffff8b8bddb72080 0 118464 2 0x00000080 [Fri Apr 12 11:12:19 2019][2937661.419697] Call Trace: [Fri Apr 12 11:12:19 2019][2937661.422331] [] schedule+0x29/0x70 [Fri Apr 12 11:12:19 2019][2937661.427523] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.434403] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:19 2019][2937661.440195] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.446679] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.454250] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:19 2019][2937661.459310] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.465587] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:19 2019][2937661.472217] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.478489] mdt03_048 S ffff8b8bddb730c0 0 118465 2 0x00000080 [Fri Apr 12 11:12:19 2019][2937661.485851] Call Trace: [Fri Apr 12 11:12:19 2019][2937661.488524] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.496282] [] schedule+0x29/0x70 [Fri Apr 12 11:12:19 2019][2937661.501466] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.508556] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.515672] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:19 2019][2937661.521462] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.528727] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:19 2019][2937661.536081] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.542912] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.550949] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:19 2019][2937661.557584] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:19 2019][2937661.564305] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:19 2019][2937661.570702] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:19 2019][2937661.577793] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:19 2019][2937661.585494] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:19 2019][2937661.592813] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:19 2019][2937661.599111] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:19 2019][2937661.606532] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:19 2019][2937661.613461] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.619886] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.627103] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.634083] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.640751] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:19 2019][2937661.648440] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:19 2019][2937661.654739] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:19 2019][2937661.660680] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:19 2019][2937661.667245] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:19 2019][2937661.673454] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.680171] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:19 2019][2937661.687165] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:19 2019][2937661.693077] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.700181] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.707945] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:19 2019][2937661.715219] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.723110] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.730091] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:19 2019][2937661.736539] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:19 2019][2937661.742604] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.749119] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.756722] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:19 2019][2937661.761796] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.768098] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:19 2019][2937661.774729] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.781000] mdt03_049 S ffff8b8bddb74100 0 118466 2 0x00000080 [Fri Apr 12 11:12:19 2019][2937661.788371] Call Trace: [Fri Apr 12 11:12:19 2019][2937661.791004] [] schedule+0x29/0x70 [Fri Apr 12 11:12:19 2019][2937661.796196] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.803087] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:19 2019][2937661.808884] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.815364] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.822937] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:19 2019][2937661.828019] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.834313] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:19 2019][2937661.840950] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:19 2019][2937661.847240] mdt02_063 S ffff8b8bddb75140 0 118467 2 0x00000080 [Fri Apr 12 11:12:19 2019][2937661.854611] Call Trace: [Fri Apr 12 11:12:19 2019][2937661.857264] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:19 2019][2937661.864189] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:19 2019][2937661.871960] [] schedule+0x29/0x70 [Fri Apr 12 11:12:20 2019][2937661.877142] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.884228] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.891312] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:20 2019][2937661.898474] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:20 2019][2937661.904296] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.911677] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.919726] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:20 2019][2937661.926450] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:20 2019][2937661.933434] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:20 2019][2937661.940205] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.948276] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.955376] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.962444] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:20 2019][2937661.969594] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:20 2019][2937661.976223] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:20 2019][2937661.983427] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.990103] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937661.997226] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.004395] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:20 2019][2937662.011117] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:20 2019][2937662.017743] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.024573] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.031464] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.038783] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.045356] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.052622] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.060330] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.066650] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.073746] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.081513] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.088803] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.096673] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.103642] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:20 2019][2937662.110099] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:20 2019][2937662.116199] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.122701] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.130303] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:20 2019][2937662.135373] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:20 2019][2937662.141680] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:20 2019][2937662.148305] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:20 2019][2937662.154573] mdt00_086 S ffff8b8bddb76180 0 118468 2 0x00000080 [Fri Apr 12 11:12:20 2019][2937662.161958] Call Trace: [Fri Apr 12 11:12:20 2019][2937662.164616] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.171496] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:20 2019][2937662.177635] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.185377] [] schedule+0x29/0x70 [Fri Apr 12 11:12:20 2019][2937662.190560] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.197649] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.204732] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:20 2019][2937662.211871] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:20 2019][2937662.217660] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.225011] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.233051] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.240806] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.247284] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.255065] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.263079] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.269727] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:20 2019][2937662.277412] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:20 2019][2937662.283698] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:20 2019][2937662.289633] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:20 2019][2937662.296179] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:20 2019][2937662.302373] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.309089] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.316067] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:20 2019][2937662.321958] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.329054] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.336810] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.344072] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.351938] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.358906] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:20 2019][2937662.365353] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:20 2019][2937662.371407] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.377887] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.385461] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:20 2019][2937662.390522] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:20 2019][2937662.396797] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:20 2019][2937662.403419] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:20 2019][2937662.409691] mdt00_087 S ffff8b8c84178000 0 118469 2 0x00000080 [Fri Apr 12 11:12:20 2019][2937662.417056] Call Trace: [Fri Apr 12 11:12:20 2019][2937662.419691] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.426612] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.434360] [] schedule+0x29/0x70 [Fri Apr 12 11:12:20 2019][2937662.439541] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.446627] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.453701] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:20 2019][2937662.460844] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:20 2019][2937662.466634] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.473983] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.482004] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.488723] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:20 2019][2937662.495696] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.502436] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.510473] [] ? check_unlink_entry+0x19/0xd0 [obdclass] [Fri Apr 12 11:12:20 2019][2937662.517624] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.524773] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:20 2019][2937662.530728] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:20 2019][2937662.537275] [] mdt_getattr_name_lock+0x11d/0x1c30 [mdt] [Fri Apr 12 11:12:20 2019][2937662.544373] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.551029] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.558089] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:20 2019][2937662.564806] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:20 2019][2937662.571436] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.578266] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.585152] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.592475] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.599048] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.606316] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.614026] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.620345] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.627438] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.635194] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.642467] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.650348] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.657314] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:20 2019][2937662.663762] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:20 2019][2937662.669817] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.676302] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.683878] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:20 2019][2937662.688940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:20 2019][2937662.695215] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:20 2019][2937662.701837] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:20 2019][2937662.708109] mdt02_064 S ffff8b8c84179040 0 118470 2 0x00000080 [Fri Apr 12 11:12:20 2019][2937662.715473] Call Trace: [Fri Apr 12 11:12:20 2019][2937662.718105] [] schedule+0x29/0x70 [Fri Apr 12 11:12:20 2019][2937662.723293] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.730174] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:20 2019][2937662.735968] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.742451] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.750023] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:20 2019][2937662.755085] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:20 2019][2937662.761360] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:20 2019][2937662.767982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:20 2019][2937662.774253] mdt00_088 S ffff8b8c8417a080 0 118471 2 0x00000080 [Fri Apr 12 11:12:20 2019][2937662.781610] Call Trace: [Fri Apr 12 11:12:20 2019][2937662.784245] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:20 2019][2937662.791168] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.798914] [] schedule+0x29/0x70 [Fri Apr 12 11:12:20 2019][2937662.804106] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.811191] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.818265] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:20 2019][2937662.825409] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:20 2019][2937662.831196] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.838544] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:20 2019][2937662.846563] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:20 2019][2937662.853285] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:21 2019][2937662.860272] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:21 2019][2937662.867018] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.875064] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.882155] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.889217] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:21 2019][2937662.896365] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:21 2019][2937662.902999] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:21 2019][2937662.910175] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.916830] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.923917] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.931072] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:21 2019][2937662.937784] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:21 2019][2937662.944415] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:21 2019][2937662.951239] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.958122] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:21 2019][2937662.965446] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:21 2019][2937662.972016] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.979293] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.987048] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937662.993372] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.000469] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.008249] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:21 2019][2937663.015531] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.023447] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.030436] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:21 2019][2937663.036901] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:21 2019][2937663.042962] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.049458] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.057059] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:21 2019][2937663.062130] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.068406] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:21 2019][2937663.075060] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.081329] mdt02_065 S ffff8b8c8417b0c0 0 118472 2 0x00000080 [Fri Apr 12 11:12:21 2019][2937663.088685] Call Trace: [Fri Apr 12 11:12:21 2019][2937663.091334] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:21 2019][2937663.098262] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.106009] [] schedule+0x29/0x70 [Fri Apr 12 11:12:21 2019][2937663.111203] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.118296] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.125382] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:21 2019][2937663.132530] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:21 2019][2937663.138334] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.145712] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.153762] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.161546] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:21 2019][2937663.168485] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.176268] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.184292] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:21 2019][2937663.190941] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:21 2019][2937663.197141] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:21 2019][2937663.203861] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:21 2019][2937663.210843] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:21 2019][2937663.216747] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.223882] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.231651] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:21 2019][2937663.238947] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.246797] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:21 2019][2937663.253250] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:21 2019][2937663.259314] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.265823] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.273400] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:21 2019][2937663.278463] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.284756] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:21 2019][2937663.291374] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.297648] mdt02_066 S ffff8b8c8417c100 0 118473 2 0x00000080 [Fri Apr 12 11:12:21 2019][2937663.305077] Call Trace: [Fri Apr 12 11:12:21 2019][2937663.307743] [] schedule+0x29/0x70 [Fri Apr 12 11:12:21 2019][2937663.312960] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.319874] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:21 2019][2937663.325692] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.332181] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.339756] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:21 2019][2937663.344824] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.351101] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:21 2019][2937663.357719] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.363993] mdt00_089 S ffff8b8c8417d140 0 118474 2 0x00000080 [Fri Apr 12 11:12:21 2019][2937663.371356] Call Trace: [Fri Apr 12 11:12:21 2019][2937663.373996] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:21 2019][2937663.380912] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.388659] [] schedule+0x29/0x70 [Fri Apr 12 11:12:21 2019][2937663.393843] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.400927] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.408015] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:21 2019][2937663.415165] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:21 2019][2937663.420987] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.428340] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.436365] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:21 2019][2937663.443081] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:21 2019][2937663.450068] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:21 2019][2937663.456826] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.464891] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.471998] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.479079] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:21 2019][2937663.486242] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:21 2019][2937663.492883] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:21 2019][2937663.500070] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.506735] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.513830] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.520979] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:21 2019][2937663.527694] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:21 2019][2937663.534346] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:21 2019][2937663.541167] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.548059] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:21 2019][2937663.555379] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:21 2019][2937663.561941] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.569197] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.576905] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.583220] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.590312] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.598061] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:21 2019][2937663.605326] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.613198] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.620161] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:21 2019][2937663.625949] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.632432] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.640006] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:21 2019][2937663.645069] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.651343] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:21 2019][2937663.657983] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.664257] mdt02_067 S ffff8b8c8417e180 0 118475 2 0x00000080 [Fri Apr 12 11:12:21 2019][2937663.671656] Call Trace: [Fri Apr 12 11:12:21 2019][2937663.674297] [] schedule+0x29/0x70 [Fri Apr 12 11:12:21 2019][2937663.679504] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.686386] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:21 2019][2937663.692221] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.698698] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.706275] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:21 2019][2937663.711346] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.717621] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:21 2019][2937663.724240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.730530] mdt02_068 S ffff8b8cda968000 0 118476 2 0x00000080 [Fri Apr 12 11:12:21 2019][2937663.737893] Call Trace: [Fri Apr 12 11:12:21 2019][2937663.740524] [] schedule+0x29/0x70 [Fri Apr 12 11:12:21 2019][2937663.745711] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.752589] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:21 2019][2937663.758378] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.764860] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.772434] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:21 2019][2937663.777497] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.783772] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:21 2019][2937663.790390] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.796656] mdt03_050 S ffff8b8cda969040 0 118477 2 0x00000080 [Fri Apr 12 11:12:21 2019][2937663.804014] Call Trace: [Fri Apr 12 11:12:21 2019][2937663.806644] [] schedule+0x29/0x70 [Fri Apr 12 11:12:21 2019][2937663.811831] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.818709] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:21 2019][2937663.824510] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.831011] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:21 2019][2937663.838609] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:21 2019][2937663.843673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:21 2019][2937663.849966] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:21 2019][2937663.856598] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937663.862900] mdt00_090 S ffff8b5c76e665a0 0 118478 2 0x00000080 [Fri Apr 12 11:12:22 2019][2937663.870272] Call Trace: [Fri Apr 12 11:12:22 2019][2937663.872954] [] ? tgt_request_handle+0xca8/0x1580 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937663.880273] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937663.888682] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:22 2019][2937663.894209] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937663.901282] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:22 2019][2937663.907072] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937663.913725] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937663.921300] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:22 2019][2937663.926532] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937663.932808] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:22 2019][2937663.939613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937663.945904] mdt02_069 S ffff8b8cda96b0c0 0 118480 2 0x00000080 [Fri Apr 12 11:12:22 2019][2937663.953274] Call Trace: [Fri Apr 12 11:12:22 2019][2937663.955909] [] schedule+0x29/0x70 [Fri Apr 12 11:12:22 2019][2937663.961094] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937663.968086] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:22 2019][2937663.973878] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937663.980367] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937663.987950] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:22 2019][2937663.993017] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937663.999306] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:22 2019][2937664.005959] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937664.012253] mdt03_051 S ffff8b8cda96c100 0 118481 2 0x00000080 [Fri Apr 12 11:12:22 2019][2937664.019610] Call Trace: [Fri Apr 12 11:12:22 2019][2937664.022243] [] schedule+0x29/0x70 [Fri Apr 12 11:12:22 2019][2937664.027434] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.034317] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:22 2019][2937664.040107] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.046589] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.054161] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:22 2019][2937664.059223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937664.065517] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:22 2019][2937664.072147] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937664.078419] mdt00_091 S ffff8b8cda96d140 0 118482 2 0x00000080 [Fri Apr 12 11:12:22 2019][2937664.085782] Call Trace: [Fri Apr 12 11:12:22 2019][2937664.088420] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:22 2019][2937664.095339] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.103086] [] schedule+0x29/0x70 [Fri Apr 12 11:12:22 2019][2937664.108264] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.115344] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.122419] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.129554] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:22 2019][2937664.135344] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.142689] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.150704] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.157422] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:22 2019][2937664.164398] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.171141] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.179180] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.186391] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.193555] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:12:22 2019][2937664.200752] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.207411] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.214513] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.221689] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:22 2019][2937664.228402] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:22 2019][2937664.235077] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.241965] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.248788] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.255675] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:22 2019][2937664.262995] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:22 2019][2937664.269560] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.276834] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.284540] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.290859] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.297954] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.305718] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:22 2019][2937664.312995] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.320872] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.327838] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:22 2019][2937664.334297] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:22 2019][2937664.340351] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.346833] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.354409] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:22 2019][2937664.359467] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937664.365744] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:22 2019][2937664.372368] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937664.378640] mdt02_070 S ffff8b8cda96e180 0 118483 2 0x00000080 [Fri Apr 12 11:12:22 2019][2937664.386011] Call Trace: [Fri Apr 12 11:12:22 2019][2937664.388680] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.396424] [] schedule+0x29/0x70 [Fri Apr 12 11:12:22 2019][2937664.401606] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.408693] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.415752] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:22 2019][2937664.421543] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.428798] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.436143] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.442981] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.451031] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:22 2019][2937664.457667] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:22 2019][2937664.464392] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:22 2019][2937664.470796] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.477868] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:22 2019][2937664.485563] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.492897] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:22 2019][2937664.499203] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:22 2019][2937664.506621] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:22 2019][2937664.513519] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.519920] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.527076] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.534058] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.540711] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.548398] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.554683] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:22 2019][2937664.560621] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:22 2019][2937664.567169] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:22 2019][2937664.573372] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.580097] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.587082] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:22 2019][2937664.593025] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.600124] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.607877] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:22 2019][2937664.615144] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.623015] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.629983] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:22 2019][2937664.636429] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:22 2019][2937664.642498] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.649007] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.656577] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:22 2019][2937664.661642] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937664.667916] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:22 2019][2937664.674548] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:22 2019][2937664.680828] mdt02_071 S ffff8b8b64a10000 0 118484 2 0x00000080 [Fri Apr 12 11:12:22 2019][2937664.688190] Call Trace: [Fri Apr 12 11:12:22 2019][2937664.690857] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.698604] [] schedule+0x29/0x70 [Fri Apr 12 11:12:22 2019][2937664.703784] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.710862] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.717914] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:22 2019][2937664.723705] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.730968] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.738325] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.745169] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.753202] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:22 2019][2937664.759845] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:22 2019][2937664.766609] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:22 2019][2937664.773006] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.780076] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:22 2019][2937664.787798] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:22 2019][2937664.795118] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:22 2019][2937664.801407] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:22 2019][2937664.808831] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:22 2019][2937664.815721] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.822124] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:22 2019][2937664.829274] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:22 2019][2937664.836256] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:23 2019][2937664.842902] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:23 2019][2937664.850586] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:23 2019][2937664.856871] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:23 2019][2937664.862810] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:23 2019][2937664.869357] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:23 2019][2937664.875554] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:23 2019][2937664.882277] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:23 2019][2937664.889250] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:23 2019][2937664.895140] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937664.902238] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937664.910011] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:23 2019][2937664.917274] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937664.925140] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937664.932106] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:23 2019][2937664.938555] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:23 2019][2937664.944609] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937664.951090] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937664.958664] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937664.963733] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937664.970017] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937664.976641] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937664.982919] mdt02_072 S ffff8b8b64a11040 0 118486 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937664.990309] Call Trace: [Fri Apr 12 11:12:23 2019][2937664.992942] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937664.998134] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.006178] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.013242] [] ? class_handle2object+0x8c/0x1c0 [obdclass] [Fri Apr 12 11:12:23 2019][2937665.020556] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.026364] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.033039] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.040619] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.045849] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.052140] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.058939] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.065229] mdt02_073 S ffff8b8b64a12080 0 118487 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.072601] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.075234] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.080425] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.087303] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:12:23 2019][2937665.094274] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.100071] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.106558] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.114148] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.119248] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.125540] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.132182] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.138455] mdt02_074 S ffff8b8b64a130c0 0 118488 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.145828] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.148486] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.153689] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.160576] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.166396] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.172878] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.180465] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.185542] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.191831] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.198449] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.204720] mdt02_075 S ffff8b8b64a14100 0 118489 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.212105] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.214760] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:23 2019][2937665.221676] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.229481] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.234670] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.241781] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.248876] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:23 2019][2937665.256016] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.261814] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.269170] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.277194] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:23 2019][2937665.283925] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:23 2019][2937665.290909] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:23 2019][2937665.297664] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.305804] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.312902] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.319982] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:23 2019][2937665.327135] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:23 2019][2937665.333765] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:23 2019][2937665.340975] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.347636] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.354722] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.361885] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:23 2019][2937665.368620] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:23 2019][2937665.375266] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:23 2019][2937665.382101] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.388990] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:23 2019][2937665.396312] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:23 2019][2937665.402895] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.410170] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.417900] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.424221] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.431316] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.439073] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:23 2019][2937665.446338] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.454210] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.461174] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:23 2019][2937665.467622] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:23 2019][2937665.473676] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.480158] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.487730] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.492790] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.499068] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.505686] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.511960] mdt_rdpg03_005 S ffff8b8b64a16180 0 118491 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.519317] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.521947] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.527132] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.534010] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.539803] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.546285] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.553857] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.558917] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.565186] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.571807] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.578082] mdt_rdpg03_006 S ffff8b7834758000 0 118492 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.585445] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.588076] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.593260] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.600138] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.605932] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.612411] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.620002] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.625064] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.631340] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.637960] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.644235] mdt_rdpg00_005 S ffff8b8b19a31040 0 118493 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.651598] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.654230] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.659417] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.666292] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.672087] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.678566] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.686141] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.691201] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.697492] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.704114] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.710389] mdt_rdpg00_006 S ffff8b8b19a36180 0 118494 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.717754] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.720385] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.725573] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.732457] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.738253] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.744742] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.752312] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.757372] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.763650] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.770270] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.776543] mdt_rdpg00_007 S ffff8b8b19a32080 0 118495 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.783906] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.786539] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.791724] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.798602] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:23 2019][2937665.804397] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.810883] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.818459] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:23 2019][2937665.823520] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.829795] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:23 2019][2937665.836414] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:23 2019][2937665.842688] mdt_rdpg03_007 S ffff8b8b19a30000 0 118496 2 0x00000080 [Fri Apr 12 11:12:23 2019][2937665.850045] Call Trace: [Fri Apr 12 11:12:23 2019][2937665.852676] [] schedule+0x29/0x70 [Fri Apr 12 11:12:23 2019][2937665.857858] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:23 2019][2937665.864740] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937665.870529] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937665.877012] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937665.884585] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937665.889645] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937665.895921] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937665.902541] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937665.908815] mdt_rdpg00_008 S ffff8b8b19a34100 0 118497 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937665.916171] Call Trace: [Fri Apr 12 11:12:24 2019][2937665.918803] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937665.923990] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937665.930884] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937665.936676] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937665.943159] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937665.950729] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937665.955791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937665.962066] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937665.968686] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937665.974954] mdt_rdpg03_008 S ffff8b8b19a35140 0 118498 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937665.982317] Call Trace: [Fri Apr 12 11:12:24 2019][2937665.984949] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937665.990135] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937665.997013] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.002803] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.009284] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.016858] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.021917] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.028185] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.034807] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.041096] mdt_rdpg03_009 S ffff8b8b19a330c0 0 118500 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.048455] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.051084] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.056273] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.063149] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.068942] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.075424] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.082997] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.088055] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.094324] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.100946] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.107218] mdt_rdpg00_009 S ffff8b8cece71040 0 118501 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.114574] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.117204] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.122393] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.129276] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.135069] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.141548] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.149124] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.154186] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.160462] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.167081] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.173356] mdt02_076 S ffff8b8cece730c0 0 118503 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.180719] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.183349] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.188536] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.195415] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.201206] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.207687] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.215261] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.220321] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.226597] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.233218] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.239491] mdt02_077 S ffff8b8cece76180 0 118504 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.246848] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.249477] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.254665] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.261543] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.267330] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.273814] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.281390] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.286450] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.292726] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.299346] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.305621] mdt_rdpg03_010 S ffff8b8cece70000 0 118505 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.312985] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.315614] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.320803] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.327680] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.333472] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.339954] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.347528] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.352586] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.358864] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.365485] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.371758] mdt_rdpg03_011 S ffff8b8cece75140 0 118506 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.379121] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.381753] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.386941] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.393825] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.399618] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.406099] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.413673] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.418731] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.425008] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.431628] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.437903] mdt02_078 S ffff8b8cece72080 0 118507 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.445267] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.447903] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:24 2019][2937666.454787] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:24 2019][2937666.460926] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.468673] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.473854] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.480942] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.488026] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:24 2019][2937666.495167] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.500956] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.508305] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.516348] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.517390] LustreError: dumping log to /tmp/lustre-log.1555092744.118307 [Fri Apr 12 11:12:24 2019][2937666.531068] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:24 2019][2937666.537551] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.545330] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.553353] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:24 2019][2937666.559998] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:24 2019][2937666.567682] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:24 2019][2937666.573966] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:24 2019][2937666.579905] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:24 2019][2937666.586452] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:24 2019][2937666.592648] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:24 2019][2937666.599363] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:24 2019][2937666.606338] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:24 2019][2937666.612227] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.619318] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.627073] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:24 2019][2937666.634333] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.642200] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.649167] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:24 2019][2937666.655614] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:24 2019][2937666.661669] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.668152] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.675722] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.680783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.687059] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.693681] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.699952] mdt02_079 S ffff8b6da99730c0 0 118508 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.707309] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.709941] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.715126] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.722003] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.727799] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.734278] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.741849] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.746902] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.753180] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.759798] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.766064] mdt_rdpg03_012 S ffff8b6da9a42080 0 118509 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.773420] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.776052] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.781239] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.788133] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.793922] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.800406] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.807977] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:24 2019][2937666.813032] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.819309] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:24 2019][2937666.825926] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:24 2019][2937666.832192] mdt02_080 S ffff8b8cd02b8000 0 118510 2 0x00000080 [Fri Apr 12 11:12:24 2019][2937666.839550] Call Trace: [Fri Apr 12 11:12:24 2019][2937666.842178] [] schedule+0x29/0x70 [Fri Apr 12 11:12:24 2019][2937666.847367] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.854245] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:24 2019][2937666.860035] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:24 2019][2937666.866515] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937666.874090] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:25 2019][2937666.879151] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937666.885426] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:25 2019][2937666.892046] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937666.898320] mdt_rdpg03_013 S ffff8b8cd02b9040 0 118511 2 0x00000080 [Fri Apr 12 11:12:25 2019][2937666.905676] Call Trace: [Fri Apr 12 11:12:25 2019][2937666.908306] [] schedule+0x29/0x70 [Fri Apr 12 11:12:25 2019][2937666.913495] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937666.920371] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:25 2019][2937666.926167] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937666.932647] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937666.940219] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:25 2019][2937666.945279] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937666.951554] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:25 2019][2937666.958175] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937666.964449] mdt_rdpg03_014 S ffff8b8cd02ba080 0 118512 2 0x00000080 [Fri Apr 12 11:12:25 2019][2937666.971821] Call Trace: [Fri Apr 12 11:12:25 2019][2937666.974452] [] schedule+0x29/0x70 [Fri Apr 12 11:12:25 2019][2937666.979639] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937666.986518] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:25 2019][2937666.992312] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937666.998792] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.006363] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:25 2019][2937667.011424] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937667.017700] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:25 2019][2937667.024321] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937667.030594] mdt02_081 S ffff8b8cd02bb0c0 0 118513 2 0x00000080 [Fri Apr 12 11:12:25 2019][2937667.037948] Call Trace: [Fri Apr 12 11:12:25 2019][2937667.040580] [] schedule+0x29/0x70 [Fri Apr 12 11:12:25 2019][2937667.045769] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.052644] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:25 2019][2937667.058437] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.064920] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.072494] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:25 2019][2937667.077554] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937667.083827] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:25 2019][2937667.090463] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937667.096731] mdt02_082 S ffff8b8cd02bc100 0 118514 2 0x00000080 [Fri Apr 12 11:12:25 2019][2937667.104086] Call Trace: [Fri Apr 12 11:12:25 2019][2937667.106725] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:25 2019][2937667.113644] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.121397] [] schedule+0x29/0x70 [Fri Apr 12 11:12:25 2019][2937667.126582] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.133668] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.140749] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.147895] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:25 2019][2937667.153687] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.161034] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.169052] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.175770] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:25 2019][2937667.182745] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.189487] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.197532] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.204625] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.211696] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.218854] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.225500] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:25 2019][2937667.232677] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.239335] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.246436] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.253589] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:25 2019][2937667.260307] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:25 2019][2937667.266936] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.273762] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.280650] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:25 2019][2937667.287964] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:25 2019][2937667.294528] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.301792] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.309498] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.315811] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.322897] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.330648] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:25 2019][2937667.337908] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.345777] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.352746] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:25 2019][2937667.359192] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:25 2019][2937667.365262] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.371746] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.379315] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:25 2019][2937667.384378] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937667.390653] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:25 2019][2937667.397274] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937667.403546] mdt02_083 S ffff8b8cd02bd140 0 118516 2 0x00000080 [Fri Apr 12 11:12:25 2019][2937667.410903] Call Trace: [Fri Apr 12 11:12:25 2019][2937667.413540] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:25 2019][2937667.420423] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:25 2019][2937667.426575] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.434319] [] schedule+0x29/0x70 [Fri Apr 12 11:12:25 2019][2937667.439499] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.446590] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.453670] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.460809] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:25 2019][2937667.466601] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.473950] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.481993] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.489754] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.496232] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.504011] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.512030] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.518674] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.526358] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.532643] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:25 2019][2937667.538580] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:25 2019][2937667.545127] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:25 2019][2937667.551322] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.558041] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.565015] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:25 2019][2937667.570903] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.577992] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.585740] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:25 2019][2937667.593002] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.600870] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.607838] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:25 2019][2937667.614283] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:25 2019][2937667.620336] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.626817] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.634391] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:25 2019][2937667.639453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937667.645727] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:25 2019][2937667.652348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:25 2019][2937667.658622] mdt02_084 S ffff8b8cd02be180 0 118517 2 0x00000080 [Fri Apr 12 11:12:25 2019][2937667.665978] Call Trace: [Fri Apr 12 11:12:25 2019][2937667.668609] [] schedule+0x29/0x70 [Fri Apr 12 11:12:25 2019][2937667.673757] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:12:25 2019][2937667.679794] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.687104] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:12:25 2019][2937667.693644] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Fri Apr 12 11:12:25 2019][2937667.700555] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.708338] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.715423] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.722480] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:25 2019][2937667.728272] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.735524] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.742874] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.749702] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.757722] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:25 2019][2937667.764349] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:25 2019][2937667.771070] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:25 2019][2937667.777454] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.784515] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:25 2019][2937667.792192] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.799512] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:25 2019][2937667.805808] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:25 2019][2937667.813246] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:25 2019][2937667.820144] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.826573] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:25 2019][2937667.833741] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.840718] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:25 2019][2937667.847363] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.855058] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:25 2019][2937667.861341] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:25 2019][2937667.867278] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:26 2019][2937667.873824] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:26 2019][2937667.880040] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:26 2019][2937667.886756] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:26 2019][2937667.893739] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:26 2019][2937667.899645] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937667.906736] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937667.914494] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:26 2019][2937667.921762] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937667.929628] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937667.936609] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:26 2019][2937667.943092] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:26 2019][2937667.949143] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937667.955637] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937667.963218] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:26 2019][2937667.968296] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937667.974569] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:26 2019][2937667.981189] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937667.987460] mdt02_085 R running task 0 118518 2 0x00000080 [Fri Apr 12 11:12:26 2019][2937667.994816] Call Trace: [Fri Apr 12 11:12:26 2019][2937667.997450] [] schedule+0x29/0x70 [Fri Apr 12 11:12:26 2019][2937668.002644] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.009549] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:26 2019][2937668.015343] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.021830] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.029404] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:26 2019][2937668.034469] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.040749] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:26 2019][2937668.047379] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.053661] mdt02_086 S ffff8b86ed701040 0 118519 2 0x00000080 [Fri Apr 12 11:12:26 2019][2937668.061024] Call Trace: [Fri Apr 12 11:12:26 2019][2937668.063665] [] schedule+0x29/0x70 [Fri Apr 12 11:12:26 2019][2937668.068866] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.075746] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:26 2019][2937668.081540] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.088028] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.095608] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:26 2019][2937668.100668] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.106965] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:26 2019][2937668.113583] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.119867] mdt_rdpg03_015 S ffff8b86ed702080 0 118521 2 0x00000080 [Fri Apr 12 11:12:26 2019][2937668.127246] Call Trace: [Fri Apr 12 11:12:26 2019][2937668.129880] [] schedule+0x29/0x70 [Fri Apr 12 11:12:26 2019][2937668.135117] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.142026] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:26 2019][2937668.147854] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.154337] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.161930] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:26 2019][2937668.166990] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.173274] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:26 2019][2937668.179912] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.186184] mdt02_087 S ffff8b86ed7030c0 0 118522 2 0x00000080 [Fri Apr 12 11:12:26 2019][2937668.193555] Call Trace: [Fri Apr 12 11:12:26 2019][2937668.196188] [] schedule+0x29/0x70 [Fri Apr 12 11:12:26 2019][2937668.201378] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.208260] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:26 2019][2937668.214052] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.220547] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.228125] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:26 2019][2937668.233187] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.239465] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:26 2019][2937668.246139] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.252420] mdt_rdpg03_016 S ffff8b86ed704100 0 118523 2 0x00000080 [Fri Apr 12 11:12:26 2019][2937668.259798] Call Trace: [Fri Apr 12 11:12:26 2019][2937668.262440] [] schedule+0x29/0x70 [Fri Apr 12 11:12:26 2019][2937668.267641] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.274521] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:26 2019][2937668.280316] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.286800] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.294378] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:26 2019][2937668.299463] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.305757] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:26 2019][2937668.312374] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.318668] mdt_rdpg03_017 S ffff8b86ed705140 0 118525 2 0x00000080 [Fri Apr 12 11:12:26 2019][2937668.326030] Call Trace: [Fri Apr 12 11:12:26 2019][2937668.328664] [] ? bit_wait+0x50/0x50 [Fri Apr 12 11:12:26 2019][2937668.334035] [] ? __radix_tree_lookup+0x84/0xf0 [Fri Apr 12 11:12:26 2019][2937668.340311] [] ? __brelse+0x3d/0x50 [Fri Apr 12 11:12:26 2019][2937668.345628] [] ? bh_lru_install+0x18a/0x1e0 [Fri Apr 12 11:12:26 2019][2937668.351649] [] ? put_dec+0x72/0x90 [Fri Apr 12 11:12:26 2019][2937668.356881] [] ? number.isra.2+0x323/0x360 [Fri Apr 12 11:12:26 2019][2937668.362816] [] ? put_dec+0x72/0x90 [Fri Apr 12 11:12:26 2019][2937668.368078] [] ? number.isra.2+0x323/0x360 [Fri Apr 12 11:12:26 2019][2937668.374019] [] ? __radix_tree_lookup+0x84/0xf0 [Fri Apr 12 11:12:26 2019][2937668.380300] [] ? __brelse+0x3d/0x50 [Fri Apr 12 11:12:26 2019][2937668.385619] [] ? bh_lru_install+0x18a/0x1e0 [Fri Apr 12 11:12:26 2019][2937668.391662] [] ? begin_wqe+0x3f/0x140 [mlx5_ib] [Fri Apr 12 11:12:26 2019][2937668.398057] [] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib] [Fri Apr 12 11:12:26 2019][2937668.405391] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:12:26 2019][2937668.410820] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Apr 12 11:12:26 2019][2937668.418748] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Apr 12 11:12:26 2019][2937668.425634] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Apr 12 11:12:26 2019][2937668.432696] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Fri Apr 12 11:12:26 2019][2937668.439420] [] ? lnet_ni_send+0x44/0xd0 [lnet] [Fri Apr 12 11:12:26 2019][2937668.445707] [] ? lnet_send+0x82/0x1c0 [lnet] [Fri Apr 12 11:12:26 2019][2937668.451845] [] ? LNetPut+0x2cc/0xb50 [lnet] [Fri Apr 12 11:12:26 2019][2937668.457902] [] ? ptl_send_buf+0x146/0x530 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.464528] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:26 2019][2937668.471777] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:12:26 2019][2937668.477459] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:12:26 2019][2937668.483226] [] ? account_entity_dequeue+0xae/0xd0 [Fri Apr 12 11:12:26 2019][2937668.489780] [] ? dequeue_entity+0x11c/0x5e0 [Fri Apr 12 11:12:26 2019][2937668.495801] [] ? set_next_entity+0x3c/0xe0 [Fri Apr 12 11:12:26 2019][2937668.501764] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:12:26 2019][2937668.507534] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:12:26 2019][2937668.514161] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:12:26 2019][2937668.520348] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:26 2019][2937668.525717] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.532776] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:26 2019][2937668.538581] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.545249] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.552823] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:26 2019][2937668.558053] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.564331] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:26 2019][2937668.571127] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.577399] mdt02_088 S ffff8b86ed706180 0 118526 2 0x00000080 [Fri Apr 12 11:12:26 2019][2937668.584760] Call Trace: [Fri Apr 12 11:12:26 2019][2937668.587411] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:26 2019][2937668.594310] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:26 2019][2937668.600501] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.608284] [] schedule+0x29/0x70 [Fri Apr 12 11:12:26 2019][2937668.613469] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.620557] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.627654] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:26 2019][2937668.634820] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:26 2019][2937668.640620] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.647973] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.656047] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.663815] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:26 2019][2937668.670314] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.678107] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.686134] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:26 2019][2937668.692798] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:26 2019][2937668.700490] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:26 2019][2937668.706776] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:26 2019][2937668.712716] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:26 2019][2937668.719264] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:26 2019][2937668.725476] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:26 2019][2937668.732191] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:26 2019][2937668.739170] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:26 2019][2937668.745077] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.752188] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.759942] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:26 2019][2937668.767222] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.775123] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.782097] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:26 2019][2937668.787888] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.794376] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.801969] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:26 2019][2937668.807029] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.813301] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:26 2019][2937668.819937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:26 2019][2937668.826203] mdt00_092 S ffff8b8c15ee0000 0 118527 2 0x00000080 [Fri Apr 12 11:12:26 2019][2937668.833596] Call Trace: [Fri Apr 12 11:12:26 2019][2937668.836238] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:26 2019][2937668.843167] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:26 2019][2937668.850919] [] schedule+0x29/0x70 [Fri Apr 12 11:12:27 2019][2937668.856103] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.863189] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.870283] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:27 2019][2937668.877448] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:27 2019][2937668.883239] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.890584] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.898636] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.906401] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:27 2019][2937668.913315] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.921100] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.929122] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:27 2019][2937668.935759] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:27 2019][2937668.941955] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:27 2019][2937668.948673] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:27 2019][2937668.955650] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:27 2019][2937668.961540] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.968640] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.976393] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:27 2019][2937668.983659] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.991535] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937668.998506] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:27 2019][2937669.004951] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:27 2019][2937669.011019] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.017532] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.025105] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:27 2019][2937669.030166] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.036459] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:27 2019][2937669.043074] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.049343] mdt_rdpg03_018 S ffff8b8c15ee1040 0 118528 2 0x00000080 [Fri Apr 12 11:12:27 2019][2937669.056706] Call Trace: [Fri Apr 12 11:12:27 2019][2937669.059337] [] schedule+0x29/0x70 [Fri Apr 12 11:12:27 2019][2937669.064549] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.071446] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:27 2019][2937669.077253] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.083815] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.091419] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:27 2019][2937669.096493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.102766] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:27 2019][2937669.109388] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.115660] mdt02_089 S ffff8b8c15ee2080 0 118529 2 0x00000080 [Fri Apr 12 11:12:27 2019][2937669.123016] Call Trace: [Fri Apr 12 11:12:27 2019][2937669.125685] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.133430] [] schedule+0x29/0x70 [Fri Apr 12 11:12:27 2019][2937669.138611] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.145699] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.152758] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:27 2019][2937669.158546] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.165776] [] ? kfree+0x106/0x140 [Fri Apr 12 11:12:27 2019][2937669.171047] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.177874] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.185893] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:27 2019][2937669.192516] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:27 2019][2937669.199230] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:27 2019][2937669.205619] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:27 2019][2937669.212677] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:27 2019][2937669.220347] [] ? mdd_object_unlock+0xe0/0xe0 [mdd] [Fri Apr 12 11:12:27 2019][2937669.226975] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:27 2019][2937669.233257] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:27 2019][2937669.240687] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:27 2019][2937669.247576] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.253976] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.261131] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.268106] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.274754] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:27 2019][2937669.282440] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:27 2019][2937669.288724] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:27 2019][2937669.294662] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:27 2019][2937669.301207] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:27 2019][2937669.307414] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.314127] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.321103] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:27 2019][2937669.326994] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.334094] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.341846] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:27 2019][2937669.349110] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.356975] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.363945] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:27 2019][2937669.370389] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:27 2019][2937669.376446] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.382932] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.390505] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:27 2019][2937669.395566] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.401844] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:27 2019][2937669.408484] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.414754] mdt02_090 S ffff8b8c15ee30c0 0 118530 2 0x00000080 [Fri Apr 12 11:12:27 2019][2937669.422116] Call Trace: [Fri Apr 12 11:12:27 2019][2937669.424749] [] schedule+0x29/0x70 [Fri Apr 12 11:12:27 2019][2937669.429938] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.436821] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:27 2019][2937669.442616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.449099] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.456670] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:27 2019][2937669.461732] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.468007] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:27 2019][2937669.474627] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.480900] mdt_rdpg03_019 S ffff8b8c15ee4100 0 118531 2 0x00000080 [Fri Apr 12 11:12:27 2019][2937669.488263] Call Trace: [Fri Apr 12 11:12:27 2019][2937669.490896] [] schedule+0x29/0x70 [Fri Apr 12 11:12:27 2019][2937669.496085] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.502967] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:27 2019][2937669.508760] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.515242] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.522812] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:27 2019][2937669.527865] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.534132] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:27 2019][2937669.540753] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.547027] mdt02_091 S ffff8b8c15ee5140 0 118532 2 0x00000080 [Fri Apr 12 11:12:27 2019][2937669.554383] Call Trace: [Fri Apr 12 11:12:27 2019][2937669.557020] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:27 2019][2937669.563942] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.571687] [] schedule+0x29/0x70 [Fri Apr 12 11:12:27 2019][2937669.576866] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.583945] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.591024] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:27 2019][2937669.598162] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:27 2019][2937669.603950] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.611292] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.619304] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.626038] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:27 2019][2937669.633014] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.639755] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.647802] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.654893] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.661954] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.669105] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.675734] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:27 2019][2937669.682909] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.689565] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.696655] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.703806] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:27 2019][2937669.710522] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:27 2019][2937669.717153] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:27 2019][2937669.723985] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.730871] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:27 2019][2937669.738192] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:27 2019][2937669.744757] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.752022] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.759727] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.766045] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.773137] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.780894] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:27 2019][2937669.788160] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.796031] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.802996] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:27 2019][2937669.808789] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.815271] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:27 2019][2937669.822843] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:27 2019][2937669.827904] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.834180] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:27 2019][2937669.840800] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:27 2019][2937669.847074] mdt02_092 S ffff8b8c15ee6180 0 118533 2 0x00000080 [Fri Apr 12 11:12:27 2019][2937669.854431] Call Trace: [Fri Apr 12 11:12:27 2019][2937669.857061] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937669.862248] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937669.869125] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937669.874918] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937669.881398] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937669.888971] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937669.894032] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937669.900306] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937669.906929] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937669.913202] mdt02_093 S ffff8b72d42b9540 0 118534 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937669.920558] Call Trace: [Fri Apr 12 11:12:28 2019][2937669.923238] [] ? tgt_request_handle+0xca8/0x1580 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937669.930511] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937669.938261] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937669.943619] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937669.950672] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937669.956483] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937669.963142] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937669.970720] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937669.975954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937669.982230] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937669.989022] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937669.995296] mdt_rdpg03_020 S ffff8b8ca8a51040 0 118535 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.002661] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.005290] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.010482] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.017363] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.023159] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.029648] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.037221] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.042281] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.048554] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.055175] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.061454] mdt00_093 S ffff8b8ca8a52080 0 118536 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.068815] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.071448] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.076630] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.083526] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.089316] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.095800] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.103374] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.108436] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.114711] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.121330] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.127605] mdt00_094 S ffff8b8ca8a530c0 0 118537 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.134968] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.137599] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.142788] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.149663] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.155460] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.161945] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.169520] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.174581] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.180865] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.187487] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.193760] mdt_rdpg03_021 S ffff8b8ca8a54100 0 118538 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.201141] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.203770] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.208971] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.215863] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.221653] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.228135] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.235708] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.240769] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.247047] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.253665] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.259940] mdt02_094 S ffff8b8ca8a55140 0 118539 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.267304] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.269946] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.275133] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.282014] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.287806] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.294297] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.301871] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.306929] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.313209] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.319855] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.326128] mdt_rdpg03_022 S ffff8b8ca8a56180 0 118540 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.333504] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.336141] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.341325] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.348209] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.354010] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.360511] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.368086] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.373145] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.379424] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.386044] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.392316] mdt02_095 S ffff8b8cbc218000 0 118541 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.399690] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.402322] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.407507] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.414385] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.420212] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.426692] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.434267] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.439346] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.445619] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.452239] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.458515] mdt_rdpg03_023 S ffff8b8cbc219040 0 118542 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.465887] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.468518] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.473704] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.480583] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.486373] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.492855] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.500428] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.505487] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.511756] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.518377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.524650] mdt03_052 S ffff8b8cbc21a080 0 118543 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.532015] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.534653] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:28 2019][2937670.541579] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.549329] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.554512] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.561609] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.568690] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:28 2019][2937670.575832] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.581620] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.588969] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.596990] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:28 2019][2937670.603705] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:28 2019][2937670.610680] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:28 2019][2937670.617426] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.617428] LustreError: dumping log to /tmp/lustre-log.1555092748.118762 [Fri Apr 12 11:12:28 2019][2937670.632419] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:28 2019][2937670.639602] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.646755] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:12:28 2019][2937670.653933] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.660604] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.667691] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.674852] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:28 2019][2937670.681567] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:28 2019][2937670.688228] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.695123] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:28 2019][2937670.701959] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.708851] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:28 2019][2937670.716172] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:28 2019][2937670.722741] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.730006] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.737720] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.744036] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.751132] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.758890] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:28 2019][2937670.766158] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.774027] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.780994] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:28 2019][2937670.787442] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:28 2019][2937670.793495] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.799981] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.807557] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:28 2019][2937670.812617] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.818895] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:28 2019][2937670.825513] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:28 2019][2937670.831786] mdt_rdpg03_024 S ffff8b8cbc21b0c0 0 118544 2 0x00000080 [Fri Apr 12 11:12:28 2019][2937670.839159] Call Trace: [Fri Apr 12 11:12:28 2019][2937670.841793] [] schedule+0x29/0x70 [Fri Apr 12 11:12:28 2019][2937670.846978] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:28 2019][2937670.853872] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:28 2019][2937670.859682] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937670.866160] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937670.873737] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:29 2019][2937670.878799] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937670.885073] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:29 2019][2937670.891692] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937670.897992] mdt03_053 S ffff8b8cbc21c100 0 118545 2 0x00000080 [Fri Apr 12 11:12:29 2019][2937670.905347] Call Trace: [Fri Apr 12 11:12:29 2019][2937670.907981] [] schedule+0x29/0x70 [Fri Apr 12 11:12:29 2019][2937670.913169] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937670.920053] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:29 2019][2937670.925844] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937670.932326] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937670.939900] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:29 2019][2937670.944969] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937670.951245] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:29 2019][2937670.957882] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937670.964158] mdt03_054 S ffff8b8cbc21d140 0 118546 2 0x00000080 [Fri Apr 12 11:12:29 2019][2937670.971563] Call Trace: [Fri Apr 12 11:12:29 2019][2937670.974195] [] schedule+0x29/0x70 [Fri Apr 12 11:12:29 2019][2937670.979383] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937670.986268] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:29 2019][2937670.992064] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937670.998552] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.006122] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:29 2019][2937671.011181] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.017449] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:29 2019][2937671.024071] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.030345] mdt00_095 S ffff8b8cbc21e180 0 118547 2 0x00000080 [Fri Apr 12 11:12:29 2019][2937671.037707] Call Trace: [Fri Apr 12 11:12:29 2019][2937671.040338] [] schedule+0x29/0x70 [Fri Apr 12 11:12:29 2019][2937671.045529] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.052412] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:29 2019][2937671.058203] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.064685] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.072260] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:29 2019][2937671.077320] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.083595] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:29 2019][2937671.090214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.096483] mdt00_096 S ffff8b8cfb610000 0 118548 2 0x00000080 [Fri Apr 12 11:12:29 2019][2937671.103845] Call Trace: [Fri Apr 12 11:12:29 2019][2937671.106484] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:29 2019][2937671.113402] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.121149] [] schedule+0x29/0x70 [Fri Apr 12 11:12:29 2019][2937671.126331] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.133421] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.140505] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:29 2019][2937671.147643] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:29 2019][2937671.153432] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.160789] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.168802] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.175519] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:29 2019][2937671.182496] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.189238] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.197283] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.204376] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.211444] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.218595] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.225228] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:29 2019][2937671.232410] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.239068] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.246163] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.253311] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:29 2019][2937671.260029] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:29 2019][2937671.266659] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.273492] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.280384] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:29 2019][2937671.287705] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:29 2019][2937671.294272] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.301534] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.309242] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.315554] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.322649] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.330398] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:29 2019][2937671.337657] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.345526] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.352496] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:29 2019][2937671.358941] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:29 2019][2937671.364993] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.371476] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.379046] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:29 2019][2937671.384100] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.390377] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:29 2019][2937671.396996] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.403270] mdt00_097 S ffff8b8cfb611040 0 118549 2 0x00000080 [Fri Apr 12 11:12:29 2019][2937671.410627] Call Trace: [Fri Apr 12 11:12:29 2019][2937671.413258] [] schedule+0x29/0x70 [Fri Apr 12 11:12:29 2019][2937671.418444] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.425323] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:29 2019][2937671.431768] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:29 2019][2937671.437568] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.444226] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.451806] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:29 2019][2937671.457041] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.463318] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:29 2019][2937671.470110] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.476400] mdt03_055 S ffff8b8cfb612080 0 118550 2 0x00000080 [Fri Apr 12 11:12:29 2019][2937671.483757] Call Trace: [Fri Apr 12 11:12:29 2019][2937671.486387] [] schedule+0x29/0x70 [Fri Apr 12 11:12:29 2019][2937671.491580] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.498461] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:29 2019][2937671.504255] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.510736] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.518309] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:29 2019][2937671.523369] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.529644] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:29 2019][2937671.536264] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.542539] mdt03_056 S ffff8b8cfb6130c0 0 118551 2 0x00000080 [Fri Apr 12 11:12:29 2019][2937671.549902] Call Trace: [Fri Apr 12 11:12:29 2019][2937671.552543] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:29 2019][2937671.559459] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.567206] [] schedule+0x29/0x70 [Fri Apr 12 11:12:29 2019][2937671.572391] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.579479] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.586570] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:29 2019][2937671.593709] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:29 2019][2937671.599504] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.606856] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.614901] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.622664] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:29 2019][2937671.629582] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.637363] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.645386] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:29 2019][2937671.652017] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:29 2019][2937671.658221] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.664939] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.671927] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:29 2019][2937671.677817] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.684909] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.692667] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:29 2019][2937671.699934] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.707802] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.714769] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:29 2019][2937671.721214] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:29 2019][2937671.727268] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.733751] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.741323] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:29 2019][2937671.746383] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.752651] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:29 2019][2937671.759273] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:29 2019][2937671.765546] mdt00_098 S ffff8b8cfb614100 0 118552 2 0x00000080 [Fri Apr 12 11:12:29 2019][2937671.772902] Call Trace: [Fri Apr 12 11:12:29 2019][2937671.775539] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:29 2019][2937671.782460] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.790205] [] schedule+0x29/0x70 [Fri Apr 12 11:12:29 2019][2937671.795393] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.802492] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.809576] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:29 2019][2937671.816717] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:29 2019][2937671.822508] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.829852] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:29 2019][2937671.837867] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.844585] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:29 2019][2937671.851561] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:29 2019][2937671.858299] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.866350] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.873464] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.880524] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:30 2019][2937671.887675] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:30 2019][2937671.894306] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:30 2019][2937671.901486] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.908147] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.915237] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.922383] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:30 2019][2937671.929101] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:30 2019][2937671.935761] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.942660] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:30 2019][2937671.949484] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.956372] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:30 2019][2937671.963689] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:30 2019][2937671.970258] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.977532] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.985238] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.991552] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937671.998649] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.006397] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:30 2019][2937672.013665] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.021535] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.028502] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:30 2019][2937672.034293] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.040775] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.048349] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:30 2019][2937672.053406] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.059676] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:30 2019][2937672.066296] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.072570] mdt00_099 S ffff8b8cfb615140 0 118553 2 0x00000080 [Fri Apr 12 11:12:30 2019][2937672.079934] Call Trace: [Fri Apr 12 11:12:30 2019][2937672.082565] [] schedule+0x29/0x70 [Fri Apr 12 11:12:30 2019][2937672.087753] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.094629] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:30 2019][2937672.100420] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.106901] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.114474] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:30 2019][2937672.119535] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.125804] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:30 2019][2937672.132423] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.138711] mdt00_100 S ffff8b8cfb616180 0 118554 2 0x00000080 [Fri Apr 12 11:12:30 2019][2937672.146071] Call Trace: [Fri Apr 12 11:12:30 2019][2937672.148702] [] schedule+0x29/0x70 [Fri Apr 12 11:12:30 2019][2937672.153889] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.160764] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:30 2019][2937672.166559] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.173039] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.180612] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:30 2019][2937672.185672] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.191941] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:30 2019][2937672.198575] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.204846] mdt00_101 R running task 0 118555 2 0x00000080 [Fri Apr 12 11:12:30 2019][2937672.212208] Call Trace: [Fri Apr 12 11:12:30 2019][2937672.214863] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.222190] [] ? osd_trans_stop+0x2d8/0x8e0 [osd_ldiskfs] [Fri Apr 12 11:12:30 2019][2937672.229451] [] ? ldlm_reprocess_all+0x10/0x20 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.236439] [] ? begin_wqe+0x3f/0x140 [mlx5_ib] [Fri Apr 12 11:12:30 2019][2937672.242804] [] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib] [Fri Apr 12 11:12:30 2019][2937672.250122] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:12:30 2019][2937672.255533] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Apr 12 11:12:30 2019][2937672.263469] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Apr 12 11:12:30 2019][2937672.270351] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Apr 12 11:12:30 2019][2937672.277407] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Fri Apr 12 11:12:30 2019][2937672.284122] [] ? lnet_ni_send+0x44/0xd0 [lnet] [Fri Apr 12 11:12:30 2019][2937672.290405] [] ? lnet_send+0x82/0x1c0 [lnet] [Fri Apr 12 11:12:30 2019][2937672.296515] [] ? LNetPut+0x2cc/0xb50 [lnet] [Fri Apr 12 11:12:30 2019][2937672.302572] [] ? ptl_send_buf+0x146/0x530 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.309200] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:30 2019][2937672.316421] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:12:30 2019][2937672.322093] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:12:30 2019][2937672.327860] [] ? account_entity_dequeue+0xae/0xd0 [Fri Apr 12 11:12:30 2019][2937672.334388] [] ? dequeue_entity+0x11c/0x5e0 [Fri Apr 12 11:12:30 2019][2937672.340401] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:12:30 2019][2937672.346671] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:12:30 2019][2937672.352598] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:12:30 2019][2937672.358352] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:12:30 2019][2937672.364973] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:12:30 2019][2937672.371164] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:30 2019][2937672.376526] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.383582] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:30 2019][2937672.389375] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.396031] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.403602] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:30 2019][2937672.408836] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.415112] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:30 2019][2937672.421907] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.428179] mdt03_057 S ffff8b8964791040 0 118556 2 0x00000080 [Fri Apr 12 11:12:30 2019][2937672.435536] Call Trace: [Fri Apr 12 11:12:30 2019][2937672.438204] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.445949] [] schedule+0x29/0x70 [Fri Apr 12 11:12:30 2019][2937672.451131] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.458224] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.465278] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:30 2019][2937672.471065] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.478323] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.485670] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.492500] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.500516] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:30 2019][2937672.507144] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:30 2019][2937672.513863] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:30 2019][2937672.520251] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.527310] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:30 2019][2937672.534999] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.542323] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:30 2019][2937672.548612] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:30 2019][2937672.556020] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:30 2019][2937672.562912] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:30 2019][2937672.569310] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.576457] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:30 2019][2937672.583434] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:30 2019][2937672.590099] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.597784] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.604068] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:30 2019][2937672.610005] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:30 2019][2937672.616550] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:30 2019][2937672.622747] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:30 2019][2937672.629465] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:30 2019][2937672.636440] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:30 2019][2937672.642328] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.649422] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.657176] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:30 2019][2937672.664445] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.672312] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.679279] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:30 2019][2937672.685724] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:30 2019][2937672.691779] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.698261] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.705831] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:30 2019][2937672.710894] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.717170] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:30 2019][2937672.723792] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:30 2019][2937672.730065] mdt00_102 S ffff8b8964792080 0 118557 2 0x00000080 [Fri Apr 12 11:12:30 2019][2937672.737428] Call Trace: [Fri Apr 12 11:12:30 2019][2937672.740093] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.747841] [] schedule+0x29/0x70 [Fri Apr 12 11:12:30 2019][2937672.753023] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.760112] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.767159] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:30 2019][2937672.772948] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.780205] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.787553] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.794382] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:30 2019][2937672.802400] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:30 2019][2937672.809027] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:30 2019][2937672.815747] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:30 2019][2937672.822133] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.829194] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:30 2019][2937672.836874] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:30 2019][2937672.844198] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:30 2019][2937672.850483] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:30 2019][2937672.857910] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:30 2019][2937672.864803] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:31 2019][2937672.871203] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937672.878349] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:31 2019][2937672.885326] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:31 2019][2937672.891974] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:31 2019][2937672.899654] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:31 2019][2937672.905935] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:31 2019][2937672.911873] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:31 2019][2937672.918417] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:31 2019][2937672.924626] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:31 2019][2937672.931337] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:31 2019][2937672.938315] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:31 2019][2937672.944205] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937672.951302] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937672.959060] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:31 2019][2937672.966330] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937672.974193] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937672.981164] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:31 2019][2937672.987611] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:31 2019][2937672.993663] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.000145] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.007718] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:31 2019][2937673.012778] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.019054] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:31 2019][2937673.025673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.031939] mdt00_103 S ffff8b89647930c0 0 118558 2 0x00000080 [Fri Apr 12 11:12:31 2019][2937673.039294] Call Trace: [Fri Apr 12 11:12:31 2019][2937673.041927] [] schedule+0x29/0x70 [Fri Apr 12 11:12:31 2019][2937673.047116] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.054013] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:31 2019][2937673.059800] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.066285] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.073855] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:31 2019][2937673.078916] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.085191] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:31 2019][2937673.091813] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.098086] mdt00_104 S ffff8b8964794100 0 118559 2 0x00000080 [Fri Apr 12 11:12:31 2019][2937673.105448] Call Trace: [Fri Apr 12 11:12:31 2019][2937673.108078] [] schedule+0x29/0x70 [Fri Apr 12 11:12:31 2019][2937673.113271] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.120152] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:31 2019][2937673.125944] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.132428] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.139999] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:31 2019][2937673.145061] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.151337] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:31 2019][2937673.157957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.164231] mdt00_105 S ffff8b8964795140 0 118560 2 0x00000080 [Fri Apr 12 11:12:31 2019][2937673.171595] Call Trace: [Fri Apr 12 11:12:31 2019][2937673.174235] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:31 2019][2937673.181154] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.188900] [] schedule+0x29/0x70 [Fri Apr 12 11:12:31 2019][2937673.194081] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.201168] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.208252] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:31 2019][2937673.215393] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:31 2019][2937673.221181] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.228535] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.236572] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.244333] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:31 2019][2937673.251249] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.259030] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.267051] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:31 2019][2937673.273681] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:31 2019][2937673.279877] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:31 2019][2937673.286594] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:31 2019][2937673.293572] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:31 2019][2937673.299459] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.306561] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.314329] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:31 2019][2937673.321593] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.329460] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.336428] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:31 2019][2937673.342877] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:31 2019][2937673.348929] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.355410] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.362979] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:31 2019][2937673.368040] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.374317] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:31 2019][2937673.380940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.387210] mdt02_096 S ffff8b8964796180 0 118561 2 0x00000080 [Fri Apr 12 11:12:31 2019][2937673.394568] Call Trace: [Fri Apr 12 11:12:31 2019][2937673.397197] [] schedule+0x29/0x70 [Fri Apr 12 11:12:31 2019][2937673.402386] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.409263] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:31 2019][2937673.415051] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.421540] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.429115] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:31 2019][2937673.434171] [] ? finish_task_switch+0xa9/0x1c0 [Fri Apr 12 11:12:31 2019][2937673.440445] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.446721] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:31 2019][2937673.453339] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.459604] mdt03_058 S ffff8b8b0ee20000 0 118562 2 0x00000080 [Fri Apr 12 11:12:31 2019][2937673.466961] Call Trace: [Fri Apr 12 11:12:31 2019][2937673.469591] [] schedule+0x29/0x70 [Fri Apr 12 11:12:31 2019][2937673.474777] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.481656] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:31 2019][2937673.487450] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.493932] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.501502] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:31 2019][2937673.506563] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.512830] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:31 2019][2937673.519452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.525725] mdt00_106 S ffff8b8b0ee21040 0 118563 2 0x00000080 [Fri Apr 12 11:12:31 2019][2937673.533080] Call Trace: [Fri Apr 12 11:12:31 2019][2937673.535712] [] schedule+0x29/0x70 [Fri Apr 12 11:12:31 2019][2937673.540900] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.547773] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:31 2019][2937673.553563] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.560055] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.567631] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:31 2019][2937673.572691] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.578968] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:31 2019][2937673.585587] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:31 2019][2937673.591854] mdt00_107 S ffff8b8b0ee22080 0 118564 2 0x00000080 [Fri Apr 12 11:12:31 2019][2937673.599218] Call Trace: [Fri Apr 12 11:12:31 2019][2937673.601886] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.609634] [] schedule+0x29/0x70 [Fri Apr 12 11:12:31 2019][2937673.614812] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.621918] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.628966] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:31 2019][2937673.634758] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.642013] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:31 2019][2937673.649362] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.656188] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.664207] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:31 2019][2937673.670836] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:31 2019][2937673.677554] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:31 2019][2937673.683942] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:31 2019][2937673.691003] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:31 2019][2937673.698689] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:31 2019][2937673.706007] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:31 2019][2937673.712293] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:31 2019][2937673.719715] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:31 2019][2937673.726605] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:31 2019][2937673.733034] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.740186] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:31 2019][2937673.747179] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:31 2019][2937673.753850] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:31 2019][2937673.761552] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:31 2019][2937673.767838] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:31 2019][2937673.773777] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:31 2019][2937673.780322] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:31 2019][2937673.786516] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:31 2019][2937673.793241] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:31 2019][2937673.800225] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:31 2019][2937673.806134] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.813252] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.821006] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:31 2019][2937673.828288] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.836160] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.843176] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:31 2019][2937673.849654] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:31 2019][2937673.855707] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:31 2019][2937673.862197] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937673.869769] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:32 2019][2937673.874828] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937673.881106] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:32 2019][2937673.887738] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937673.894048] mdt02_097 R running task 0 118565 2 0x00000080 [Fri Apr 12 11:12:32 2019][2937673.901426] Call Trace: [Fri Apr 12 11:12:32 2019][2937673.904066] [] schedule+0x29/0x70 [Fri Apr 12 11:12:32 2019][2937673.909257] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937673.917311] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937673.924295] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:32 2019][2937673.930768] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:32 2019][2937673.936576] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937673.943236] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937673.950859] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:32 2019][2937673.956118] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937673.962411] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:32 2019][2937673.969203] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937673.975503] mdt_rdpg03_025 S ffff8b8b0ee24100 0 118566 2 0x00000080 [Fri Apr 12 11:12:32 2019][2937673.982865] Call Trace: [Fri Apr 12 11:12:32 2019][2937673.985498] [] schedule+0x29/0x70 [Fri Apr 12 11:12:32 2019][2937673.990690] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937673.997606] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:12:32 2019][2937674.004593] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:32 2019][2937674.010384] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.016864] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.024438] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:32 2019][2937674.029503] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.035785] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:32 2019][2937674.042414] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.048682] mdt00_108 S ffff8b8b0ee25140 0 118567 2 0x00000080 [Fri Apr 12 11:12:32 2019][2937674.056038] Call Trace: [Fri Apr 12 11:12:32 2019][2937674.058672] [] schedule+0x29/0x70 [Fri Apr 12 11:12:32 2019][2937674.063860] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.070743] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:32 2019][2937674.076578] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.083094] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.090691] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:32 2019][2937674.095747] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.102041] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:32 2019][2937674.108689] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.114980] mdt02_098 S ffff8b8b0ee26180 0 118568 2 0x00000080 [Fri Apr 12 11:12:32 2019][2937674.122386] Call Trace: [Fri Apr 12 11:12:32 2019][2937674.125066] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.132823] [] schedule+0x29/0x70 [Fri Apr 12 11:12:32 2019][2937674.138014] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.145118] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.152171] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:32 2019][2937674.157976] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.165258] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:32 2019][2937674.172614] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.179454] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.187491] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:32 2019][2937674.194124] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:32 2019][2937674.200842] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:32 2019][2937674.207232] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:32 2019][2937674.214291] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:32 2019][2937674.221976] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:32 2019][2937674.229307] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:32 2019][2937674.235594] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:32 2019][2937674.243043] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:32 2019][2937674.249946] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.256370] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.263538] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.270515] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.277166] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:32 2019][2937674.284848] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:32 2019][2937674.291144] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:32 2019][2937674.297093] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:32 2019][2937674.303637] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:32 2019][2937674.309867] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.316603] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.323601] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:32 2019][2937674.329499] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.336609] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.344376] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:32 2019][2937674.351651] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.359530] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.366547] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:32 2019][2937674.373007] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:32 2019][2937674.379083] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.385588] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.393180] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:32 2019][2937674.398238] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.404512] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:32 2019][2937674.411133] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.417406] mdt00_109 S ffff8b7da9558000 0 118569 2 0x00000080 [Fri Apr 12 11:12:32 2019][2937674.424765] Call Trace: [Fri Apr 12 11:12:32 2019][2937674.427393] [] schedule+0x29/0x70 [Fri Apr 12 11:12:32 2019][2937674.432587] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.439465] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:32 2019][2937674.445258] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.451740] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.459312] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:32 2019][2937674.464374] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.470648] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:32 2019][2937674.477281] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.483565] mdt_rdpg03_026 S ffff8b7da9559040 0 118570 2 0x00000080 [Fri Apr 12 11:12:32 2019][2937674.490945] Call Trace: [Fri Apr 12 11:12:32 2019][2937674.493577] [] schedule+0x29/0x70 [Fri Apr 12 11:12:32 2019][2937674.498769] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.505658] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:32 2019][2937674.511449] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.517932] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.525513] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:32 2019][2937674.530573] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.536851] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:32 2019][2937674.543469] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.549741] mdt03_059 S ffff8b7da955a080 0 118571 2 0x00000080 [Fri Apr 12 11:12:32 2019][2937674.557132] Call Trace: [Fri Apr 12 11:12:32 2019][2937674.559774] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:32 2019][2937674.566692] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.574436] [] schedule+0x29/0x70 [Fri Apr 12 11:12:32 2019][2937674.579625] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.586730] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.593849] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:32 2019][2937674.600996] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:32 2019][2937674.606816] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.614189] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.622214] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.628988] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:32 2019][2937674.635964] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.642707] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.650758] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.657884] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.664962] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.672130] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.678779] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:32 2019][2937674.685967] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.692634] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.699746] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.706904] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:32 2019][2937674.713621] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:32 2019][2937674.720249] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:32 2019][2937674.727076] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.733966] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:32 2019][2937674.741287] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:32 2019][2937674.747862] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.755127] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.762833] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.769157] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.776252] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.784005] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:32 2019][2937674.791264] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.799135] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.806120] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:32 2019][2937674.812567] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:32 2019][2937674.818635] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.825118] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:32 2019][2937674.832700] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:32 2019][2937674.837761] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.844040] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:32 2019][2937674.850662] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:32 2019][2937674.856939] mdt00_110 D ffff8b7da955b0c0 0 118572 2 0x00000080 [Fri Apr 12 11:12:32 2019][2937674.864307] Call Trace: [Fri Apr 12 11:12:32 2019][2937674.866943] [] schedule_preempt_disabled+0x29/0x70 [Fri Apr 12 11:12:33 2019][2937674.873565] [] __mutex_lock_slowpath+0xc7/0x1d0 [Fri Apr 12 11:12:33 2019][2937674.879924] [] mutex_lock+0x1f/0x2f [Fri Apr 12 11:12:33 2019][2937674.885302] [] tgt_handle_received_xid+0x56/0xe0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937674.892585] [] tgt_request_handle+0xca8/0x1580 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937674.899692] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937674.907451] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:33 2019][2937674.914737] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937674.922603] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937674.929572] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:33 2019][2937674.935373] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937674.941853] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937674.949427] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:33 2019][2937674.954496] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937674.960781] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:33 2019][2937674.967402] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937674.973673] mdt00_111 S ffff8b7da955c100 0 118573 2 0x00000080 [Fri Apr 12 11:12:33 2019][2937674.981054] Call Trace: [Fri Apr 12 11:12:33 2019][2937674.983689] [] schedule+0x29/0x70 [Fri Apr 12 11:12:33 2019][2937674.988877] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937674.995761] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:33 2019][2937675.001552] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.008039] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.015635] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:33 2019][2937675.020702] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.026979] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:33 2019][2937675.033608] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.039895] mdt03_060 S ffff8b7da955d140 0 118574 2 0x00000080 [Fri Apr 12 11:12:33 2019][2937675.047259] Call Trace: [Fri Apr 12 11:12:33 2019][2937675.049893] [] schedule+0x29/0x70 [Fri Apr 12 11:12:33 2019][2937675.055095] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.061977] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:33 2019][2937675.067778] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.074263] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.081856] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:33 2019][2937675.086926] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.093213] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:33 2019][2937675.099831] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.106105] mdt02_099 S ffff8b7da955e180 0 118575 2 0x00000080 [Fri Apr 12 11:12:33 2019][2937675.113478] Call Trace: [Fri Apr 12 11:12:33 2019][2937675.116129] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:33 2019][2937675.123102] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.130871] [] schedule+0x29/0x70 [Fri Apr 12 11:12:33 2019][2937675.136062] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.143166] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.150249] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:33 2019][2937675.157391] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:33 2019][2937675.163183] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.170532] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.178554] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.185294] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:33 2019][2937675.192276] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.199052] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.207095] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.214249] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.221322] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.228506] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.235151] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:33 2019][2937675.242369] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.249031] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.256177] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.263337] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:33 2019][2937675.270075] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:33 2019][2937675.276733] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.283565] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.290484] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:33 2019][2937675.297825] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:33 2019][2937675.304402] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.311693] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.319471] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.325804] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.332897] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.340674] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:33 2019][2937675.347945] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.355846] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.362830] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:33 2019][2937675.369272] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:33 2019][2937675.375326] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.381809] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.389382] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:33 2019][2937675.394445] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.400745] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:33 2019][2937675.407369] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.413640] mdt02_100 R running task 0 118576 2 0x00000080 [Fri Apr 12 11:12:33 2019][2937675.421038] Call Trace: [Fri Apr 12 11:12:33 2019][2937675.423727] [] ? tgt_request_handle+0xca8/0x1580 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.430960] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:33 2019][2937675.436318] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.443382] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:33 2019][2937675.449202] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.455870] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.463453] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:33 2019][2937675.468682] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.474961] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:33 2019][2937675.481753] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.488024] mdt02_101 S ffff8b8bb4961040 0 118577 2 0x00000080 [Fri Apr 12 11:12:33 2019][2937675.495401] Call Trace: [Fri Apr 12 11:12:33 2019][2937675.498049] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:33 2019][2937675.504974] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.512723] [] schedule+0x29/0x70 [Fri Apr 12 11:12:33 2019][2937675.517906] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.524995] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.532074] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:33 2019][2937675.539214] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:33 2019][2937675.545002] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.552353] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.560414] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.568190] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:33 2019][2937675.575130] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.582915] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.590940] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:33 2019][2937675.597580] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:33 2019][2937675.603784] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.610505] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.617485] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:33 2019][2937675.623378] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.630472] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.638220] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:33 2019][2937675.645480] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.653350] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.660317] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:33 2019][2937675.666762] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:33 2019][2937675.672816] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.679295] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.686870] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:33 2019][2937675.691932] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.698208] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:33 2019][2937675.704830] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:33 2019][2937675.711102] mdt02_102 S ffff8b8bb4962080 0 118578 2 0x00000080 [Fri Apr 12 11:12:33 2019][2937675.718464] Call Trace: [Fri Apr 12 11:12:33 2019][2937675.721103] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:33 2019][2937675.728024] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.735771] [] schedule+0x29/0x70 [Fri Apr 12 11:12:33 2019][2937675.740952] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.748040] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.755125] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:33 2019][2937675.762264] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:33 2019][2937675.768053] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.775401] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.783422] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.790140] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:33 2019][2937675.797116] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.803857] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.811906] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.818974] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:33 2019][2937675.826147] [] ? lustre_swab_generic_32s+0x20/0x20 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.833558] [] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Fri Apr 12 11:12:33 2019][2937675.840649] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.847309] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.854396] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:33 2019][2937675.861546] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:33 2019][2937675.868264] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:33 2019][2937675.874912] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:34 2019][2937675.881739] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.888633] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:34 2019][2937675.895969] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:34 2019][2937675.902543] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.909812] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.917543] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.923858] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.930979] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.938730] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:34 2019][2937675.945998] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.953872] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.960844] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:34 2019][2937675.967298] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:34 2019][2937675.973350] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.979830] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937675.987406] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:34 2019][2937675.992467] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937675.998742] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:34 2019][2937676.005392] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.011672] mdt03_061 S ffff8b8bb49630c0 0 118579 2 0x00000080 [Fri Apr 12 11:12:34 2019][2937676.019042] Call Trace: [Fri Apr 12 11:12:34 2019][2937676.021698] [] schedule+0x29/0x70 [Fri Apr 12 11:12:34 2019][2937676.026889] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.033782] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:34 2019][2937676.039782] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.046326] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.053933] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:34 2019][2937676.059010] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.065302] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:34 2019][2937676.071922] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.078198] mdt02_103 S ffff8b8bb4964100 0 118580 2 0x00000080 [Fri Apr 12 11:12:34 2019][2937676.085560] Call Trace: [Fri Apr 12 11:12:34 2019][2937676.088254] [] ? tgt_request_handle+0xca8/0x1580 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.095532] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.103286] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:34 2019][2937676.110520] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:34 2019][2937676.115875] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.122998] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:34 2019][2937676.128791] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.135453] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.143026] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:34 2019][2937676.148261] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.154536] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:34 2019][2937676.161334] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.167603] mdt_rdpg03_027 S ffff8b8bb4965140 0 118581 2 0x00000080 [Fri Apr 12 11:12:34 2019][2937676.174968] Call Trace: [Fri Apr 12 11:12:34 2019][2937676.177598] [] schedule+0x29/0x70 [Fri Apr 12 11:12:34 2019][2937676.182789] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.189683] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:34 2019][2937676.195493] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.201979] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.209555] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:34 2019][2937676.214625] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.220923] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:34 2019][2937676.227546] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.233819] mdt02_104 S ffff8b8bb4966180 0 118582 2 0x00000080 [Fri Apr 12 11:12:34 2019][2937676.241183] Call Trace: [Fri Apr 12 11:12:34 2019][2937676.243824] [] schedule+0x29/0x70 [Fri Apr 12 11:12:34 2019][2937676.249019] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.255937] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:34 2019][2937676.261737] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.268224] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.275806] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:34 2019][2937676.280866] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.287149] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:34 2019][2937676.293815] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.300086] mdt02_105 S ffff8b8c623c8000 0 118583 2 0x00000080 [Fri Apr 12 11:12:34 2019][2937676.307468] Call Trace: [Fri Apr 12 11:12:34 2019][2937676.310108] [] schedule+0x29/0x70 [Fri Apr 12 11:12:34 2019][2937676.315301] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.322179] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:34 2019][2937676.327972] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.334481] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.342065] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:34 2019][2937676.347125] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.353397] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:34 2019][2937676.360020] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.366290] mdt02_106 S ffff8b8c623c9040 0 118584 2 0x00000080 [Fri Apr 12 11:12:34 2019][2937676.373655] Call Trace: [Fri Apr 12 11:12:34 2019][2937676.376325] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.384081] [] schedule+0x29/0x70 [Fri Apr 12 11:12:34 2019][2937676.389288] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.396374] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.403435] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:34 2019][2937676.409231] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.416495] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:34 2019][2937676.423843] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.430711] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.438773] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:34 2019][2937676.445408] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:34 2019][2937676.452132] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:34 2019][2937676.458520] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:34 2019][2937676.465589] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:34 2019][2937676.473293] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:34 2019][2937676.480620] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:34 2019][2937676.486916] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:34 2019][2937676.494333] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:34 2019][2937676.501225] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.507652] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.514829] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.521805] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.528461] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:34 2019][2937676.536155] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:34 2019][2937676.542450] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:34 2019][2937676.548396] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:34 2019][2937676.554940] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:34 2019][2937676.561135] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.567853] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.574861] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:34 2019][2937676.580764] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.587901] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.595688] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:34 2019][2937676.602956] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.610797] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:34 2019][2937676.617254] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:34 2019][2937676.623304] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.629787] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.637358] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:34 2019][2937676.642418] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.648695] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:34 2019][2937676.655314] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:34 2019][2937676.661590] mdt02_107 S ffff8b8c623ca080 0 118585 2 0x00000080 [Fri Apr 12 11:12:34 2019][2937676.668953] Call Trace: [Fri Apr 12 11:12:34 2019][2937676.671595] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:34 2019][2937676.678474] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:34 2019][2937676.684615] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.692366] [] schedule+0x29/0x70 [Fri Apr 12 11:12:34 2019][2937676.697548] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.704636] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.711722] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:34 2019][2937676.718883] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:34 2019][2937676.724668] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.732016] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.740094] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.748113] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.754603] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.762421] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.770447] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.777120] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:34 2019][2937676.784827] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:34 2019][2937676.791122] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:34 2019][2937676.797083] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:34 2019][2937676.803646] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:34 2019][2937676.809857] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.816574] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:34 2019][2937676.823586] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:34 2019][2937676.829496] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.836590] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:34 2019][2937676.844353] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:34 2019][2937676.851648] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937676.859530] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937676.866513] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:35 2019][2937676.872967] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:35 2019][2937676.879057] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937676.885547] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937676.893127] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:35 2019][2937676.898189] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:35 2019][2937676.904467] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:35 2019][2937676.911101] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:35 2019][2937676.917375] mdt03_062 S ffff8b8c623cb0c0 0 118586 2 0x00000080 [Fri Apr 12 11:12:35 2019][2937676.924798] Call Trace: [Fri Apr 12 11:12:35 2019][2937676.927439] [] schedule+0x29/0x70 [Fri Apr 12 11:12:35 2019][2937676.932645] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937676.939532] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:35 2019][2937676.945324] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937676.951820] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937676.959392] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:35 2019][2937676.964454] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:35 2019][2937676.970728] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:35 2019][2937676.977349] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:35 2019][2937676.983622] mdt02_108 S ffff8b8c623cc100 0 118587 2 0x00000080 [Fri Apr 12 11:12:35 2019][2937676.990995] Call Trace: [Fri Apr 12 11:12:35 2019][2937676.993637] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:35 2019][2937677.000556] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.008303] [] schedule+0x29/0x70 [Fri Apr 12 11:12:35 2019][2937677.013484] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.020599] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.027678] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.034823] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:35 2019][2937677.040630] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.047991] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.056018] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.062739] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:35 2019][2937677.069715] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.076495] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.084552] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.091647] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.098716] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.105866] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.112496] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:35 2019][2937677.119679] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.126341] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.133433] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.140583] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:35 2019][2937677.147300] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:35 2019][2937677.153950] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.160794] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.167683] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:35 2019][2937677.175002] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:35 2019][2937677.181568] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.188834] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.196536] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.202853] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.209948] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.217806] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:35 2019][2937677.225084] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.232966] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.239952] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:35 2019][2937677.246414] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:35 2019][2937677.252489] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.258996] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.266584] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:35 2019][2937677.271645] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:35 2019][2937677.277921] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:35 2019][2937677.284540] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:35 2019][2937677.290813] mdt02_109 S ffff8b8c623cd140 0 118588 2 0x00000080 [Fri Apr 12 11:12:35 2019][2937677.298175] Call Trace: [Fri Apr 12 11:12:35 2019][2937677.300846] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.308605] [] schedule+0x29/0x70 [Fri Apr 12 11:12:35 2019][2937677.313788] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.320879] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.327934] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:35 2019][2937677.333736] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.340997] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.348355] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.355192] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.363209] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:35 2019][2937677.369835] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:35 2019][2937677.376554] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:35 2019][2937677.382944] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.390005] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:35 2019][2937677.397689] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.405012] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:35 2019][2937677.411297] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:35 2019][2937677.418713] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:35 2019][2937677.425605] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.432034] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.439216] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.446205] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.452867] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.460551] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.466839] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:35 2019][2937677.472785] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:35 2019][2937677.479335] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:35 2019][2937677.485569] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.492305] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.499278] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:35 2019][2937677.505176] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.512278] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.520044] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:35 2019][2937677.527310] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.535225] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.542218] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:35 2019][2937677.548688] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:35 2019][2937677.554761] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.561248] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.568823] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:35 2019][2937677.573887] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:35 2019][2937677.580169] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:35 2019][2937677.586815] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:35 2019][2937677.593103] mdt03_063 S ffff8b8c623ce180 0 118589 2 0x00000080 [Fri Apr 12 11:12:35 2019][2937677.600494] Call Trace: [Fri Apr 12 11:12:35 2019][2937677.603171] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.610931] [] schedule+0x29/0x70 [Fri Apr 12 11:12:35 2019][2937677.616131] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.623239] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.630307] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:35 2019][2937677.636104] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.643391] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.650753] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.657580] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.665602] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:35 2019][2937677.672242] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:35 2019][2937677.678972] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:35 2019][2937677.685372] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.692446] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:35 2019][2937677.700154] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.707495] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:35 2019][2937677.713788] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:35 2019][2937677.721220] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:35 2019][2937677.728132] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.734565] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.741757] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.748740] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.755404] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.763124] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:35 2019][2937677.769411] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:35 2019][2937677.775355] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:35 2019][2937677.781945] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:35 2019][2937677.788149] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.794872] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:35 2019][2937677.801849] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:35 2019][2937677.807755] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.814859] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.822624] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:35 2019][2937677.829894] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.837769] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:35 2019][2937677.844759] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:35 2019][2937677.851205] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:35 2019][2937677.857272] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937677.863756] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937677.871340] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937677.876401] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937677.882685] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937677.889316] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937677.895595] mdt_rdpg03_028 S ffff8b8c82018000 0 118590 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937677.902969] Call Trace: [Fri Apr 12 11:12:36 2019][2937677.905612] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937677.910827] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937677.917727] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:12:36 2019][2937677.924712] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937677.930516] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937677.937000] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937677.944593] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937677.949662] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937677.955936] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937677.962559] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937677.968828] mdt02_110 S ffff8b8c82019040 0 118591 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937677.976219] Call Trace: [Fri Apr 12 11:12:36 2019][2937677.978851] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937677.984042] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937677.990923] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937677.996713] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.003198] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.010770] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.015829] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.022106] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.028725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.035011] mdt02_111 S ffff8b8c8201a080 0 118592 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.042411] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.045050] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.050264] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.057146] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.062939] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.069435] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.077012] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.082070] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.088347] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.094968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.101260] mdt03_064 S ffff8b8c8201b0c0 0 118593 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.108623] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.111252] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.116441] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.123358] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.129183] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.135681] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.143272] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.148331] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.154608] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.161235] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.167509] mdt03_065 S ffff8b8c8201c100 0 118594 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.174880] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.177514] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.182707] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.189587] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.195378] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.201867] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.209442] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.214503] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.220777] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.227400] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.233680] mdt_out01_003 S ffff8b8c8201d140 0 118595 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.241087] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.243720] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.248917] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.255808] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.261636] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.268140] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.275726] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.280787] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.287062] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.293705] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.299983] mdt_io01_007 S ffff8b8c8201e180 0 118596 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.307366] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.310006] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.315197] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.322077] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.327870] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.334363] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.341942] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.347000] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.353295] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.359936] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.366206] mdt_io02_007 S ffff8b8bf1ce0000 0 118597 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.373595] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.376237] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.381429] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.388309] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.394099] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.400581] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.408163] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.413223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.419516] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.426139] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.432412] mdt_io03_004 S ffff8b8bf1ce1040 0 118598 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.439785] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.442425] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.447616] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.454496] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.460315] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.466795] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.474375] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.479441] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.485715] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.492355] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.498644] mdt_io00_007 S ffff8b8bf1ce2080 0 118599 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.506008] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.508639] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.513850] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.520734] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.526531] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.533020] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.540594] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.545665] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.551958] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.558588] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.564870] mdt_io01_008 S ffff8b8bf1ce30c0 0 118600 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.572231] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.574863] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.580052] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.586936] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.592728] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.599208] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.606782] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.611841] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.618118] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.624741] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.631030] mdt_io02_008 S ffff8b8bf1ce4100 0 118601 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.638393] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.641025] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.646214] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.653109] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.658898] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.665405] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.672991] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.678067] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.684363] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.690981] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.697258] mdt_io03_005 S ffff8b8bf1ce5140 0 118602 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.704639] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.707276] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.712465] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.719368] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.725156] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.731636] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.739209] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.744277] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.750558] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.757180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.763454] mdt03_066 S ffff8b8bf1ce6180 0 118603 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.770816] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.773450] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.778668] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.785562] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.791372] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.797851] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.805438] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:36 2019][2937678.810531] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.816804] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:36 2019][2937678.823426] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:36 2019][2937678.829701] mdt_io00_008 S ffff8b8c32eb0000 0 118604 2 0x00000080 [Fri Apr 12 11:12:36 2019][2937678.837063] Call Trace: [Fri Apr 12 11:12:36 2019][2937678.839707] [] schedule+0x29/0x70 [Fri Apr 12 11:12:36 2019][2937678.844946] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.851830] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:36 2019][2937678.857616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:36 2019][2937678.864126] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937678.871719] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937678.876780] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937678.883067] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937678.889685] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937678.895959] mdt_io02_009 S ffff8b8c32eb1040 0 118605 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937678.903322] Call Trace: [Fri Apr 12 11:12:37 2019][2937678.905954] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937678.911141] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937678.918016] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:12:37 2019][2937678.924984] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937678.930798] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937678.937305] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937678.944885] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937678.949947] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937678.956224] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937678.962881] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937678.969159] mdt_io01_009 S ffff8b8c32eb2080 0 118606 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937678.976526] Call Trace: [Fri Apr 12 11:12:37 2019][2937678.979183] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937678.984371] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937678.991255] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:37 2019][2937678.997708] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.003504] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.010158] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.017732] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.022970] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.029268] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.036070] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.042341] mdt_io00_009 S ffff8b8c32eb30c0 0 118607 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.049706] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.052339] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.057527] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.064410] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.070201] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.076686] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.084256] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.089319] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.095599] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.102241] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.108530] mdt_io00_010 S ffff8b8c32eb4100 0 118608 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.115938] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.118574] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.123794] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.130689] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.136481] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.142975] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.150558] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.155640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.161911] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.168534] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.174823] mdt_io02_010 S ffff8b8c32eb5140 0 118609 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.182177] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.184811] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.190006] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.196887] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.202697] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.209181] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.216755] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.221817] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.228093] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.234733] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.241028] mdt03_067 S ffff8b8c32eb6180 0 118610 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.248395] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.251035] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.256240] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.263127] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.268921] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.275415] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.283011] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.288076] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.294371] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.301000] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.307270] mdt_io00_011 S ffff8b8966f68000 0 118611 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.314638] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.317274] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.322467] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.329349] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.335153] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.341644] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.349237] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.354322] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.360603] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.367221] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.373494] mdt_io03_006 S ffff8b8966f69040 0 118612 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.380857] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.383491] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.388693] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.395599] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.401391] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.407869] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.415468] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.420542] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.426816] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.433435] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.439747] mdt_io01_010 S ffff8b8966f6a080 0 118613 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.447115] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.449750] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.454943] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.461822] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.467612] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.474103] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.481677] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.486745] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.493053] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.499682] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.505950] mdt_out01_004 S ffff8b8966f6b0c0 0 118614 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.513315] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.515943] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.521135] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.528020] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.533809] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.540290] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.547868] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.552925] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.559201] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.565822] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.572098] mdt_io02_011 S ffff8b8966f6c100 0 118615 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.579458] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.582093] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.587306] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.594212] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.600011] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.606523] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.614096] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.619158] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.625432] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.632055] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.638328] mdt03_068 S ffff8b8966f6d140 0 118616 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.645691] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.648324] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.653515] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.660395] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.666189] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.672669] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.680242] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.685301] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.691577] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.698198] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.704471] mdt_io00_012 S ffff8b8966f6e180 0 118617 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.711826] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.714459] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.719650] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.726530] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.732322] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.738805] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.746379] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.751439] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.757715] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.764335] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.770608] mdt_io00_013 S ffff8b8cfe500000 0 118618 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.777964] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.780595] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.785784] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.792659] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.798448] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.804937] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.812512] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:37 2019][2937679.817577] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.823852] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:37 2019][2937679.830474] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:37 2019][2937679.836763] mdt_io02_012 S ffff8b8cfe501040 0 118619 2 0x00000080 [Fri Apr 12 11:12:37 2019][2937679.844128] Call Trace: [Fri Apr 12 11:12:37 2019][2937679.846760] [] schedule+0x29/0x70 [Fri Apr 12 11:12:37 2019][2937679.851951] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:37 2019][2937679.858831] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:37 2019][2937679.864623] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937679.871106] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937679.878677] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937679.883740] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937679.890016] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937679.896636] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937679.902910] mdt_io03_007 S ffff8b8cfe502080 0 118620 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937679.910273] Call Trace: [Fri Apr 12 11:12:38 2019][2937679.912901] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937679.918087] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937679.924973] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937679.930762] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937679.937244] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937679.944830] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937679.949903] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937679.956181] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937679.962807] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937679.969099] mdt_io00_014 S ffff8b8cfe5030c0 0 118621 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937679.976460] Call Trace: [Fri Apr 12 11:12:38 2019][2937679.979094] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937679.984287] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937679.991192] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937679.996984] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.003465] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.011059] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.016117] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.022393] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.029022] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.035296] mdt03_069 S ffff8b8cfe504100 0 118622 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.042660] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.045290] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.050480] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.057364] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.063151] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.069657] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.077236] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.082297] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.088582] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.095212] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.101494] mdt_io03_008 S ffff8b8cfe505140 0 118623 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.108867] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.111532] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.116724] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.123605] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.129398] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.135878] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.143453] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.148514] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.154788] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.161410] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.167682] mdt_io00_015 S ffff8b5cfb639040 0 118624 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.175045] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.177680] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.182901] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.189794] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.195586] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.202065] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.209644] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.214702] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.220978] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.227596] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.233870] mdt_io03_009 S ffff8b5cfb63b0c0 0 118625 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.241240] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.243876] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.249067] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.255947] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.261745] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.268228] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.275802] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.280866] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.287183] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.293813] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.300084] mdt_io02_013 S ffff8b5a513a30c0 0 118626 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.307450] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.310082] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.315268] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.322148] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.327935] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.334417] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.342010] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.347070] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.353347] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.359970] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.366259] mdt_io00_016 S ffff8b5a513a4100 0 118627 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.373621] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.376253] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.381440] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.388341] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.394132] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.400614] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.408187] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.413249] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.419525] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.426148] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.432419] mdt_io02_014 S ffff8b5a513a1040 0 118628 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.439776] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.442407] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.447594] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.454471] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:12:38 2019][2937680.461436] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.467220] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.473702] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.481275] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.486336] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.492611] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.499248] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.505515] mdt03_070 S ffff8b5a513a0000 0 118629 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.512872] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.515501] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.520690] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.527572] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.533357] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.539838] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.547413] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.552473] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.558762] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.565379] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.571654] mdt_io00_017 S ffff8b5a513a2080 0 118630 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.579017] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.581647] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.586834] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.593711] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.599503] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.605985] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.613580] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.618639] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.624914] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.631533] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.637808] mdt03_071 S ffff8b5a513a6180 0 118631 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.645171] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.647803] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.652992] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.659875] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.665675] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.672160] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.679730] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.684791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.691065] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.697688] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.703962] mdt_io01_011 S ffff8b5a513a5140 0 118632 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.711324] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.713957] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.719147] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.726030] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.731821] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.738306] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.745884] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.750944] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.757220] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:38 2019][2937680.763842] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:38 2019][2937680.770114] mdt_io03_010 S ffff8b8cfa710000 0 118633 2 0x00000080 [Fri Apr 12 11:12:38 2019][2937680.777470] Call Trace: [Fri Apr 12 11:12:38 2019][2937680.780100] [] schedule+0x29/0x70 [Fri Apr 12 11:12:38 2019][2937680.785287] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.792167] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:38 2019][2937680.797958] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.804440] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:38 2019][2937680.812013] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:38 2019][2937680.817073] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937680.823350] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937680.829969] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937680.836243] mdt_io02_015 S ffff8b8cfa712080 0 118634 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937680.843600] Call Trace: [Fri Apr 12 11:12:39 2019][2937680.846230] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937680.851416] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937680.858311] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937680.864102] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937680.870586] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937680.878157] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937680.883218] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937680.889495] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937680.896114] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937680.902389] mdt_io00_018 S ffff8b8cfa714100 0 118635 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937680.909743] Call Trace: [Fri Apr 12 11:12:39 2019][2937680.912376] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937680.917565] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937680.924449] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937680.930240] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937680.936725] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937680.944296] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937680.949357] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937680.955672] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937680.962294] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937680.968568] mdt03_072 S ffff8b8cfa711040 0 118636 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937680.975925] Call Trace: [Fri Apr 12 11:12:39 2019][2937680.978557] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937680.983743] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937680.990619] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937680.996411] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.002894] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.010469] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937681.015526] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.021805] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937681.028443] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.034719] mdt_io01_012 S ffff8b8cfa715140 0 118637 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937681.042089] Call Trace: [Fri Apr 12 11:12:39 2019][2937681.044721] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937681.049931] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.056813] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937681.062613] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.069115] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.076709] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937681.081772] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.088074] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937681.094763] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.101179] mdt_io02_016 S ffff8b4e294d2080 0 118638 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937681.108593] Call Trace: [Fri Apr 12 11:12:39 2019][2937681.111232] [] __cond_resched+0x26/0x30 [Fri Apr 12 11:12:39 2019][2937681.116907] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Apr 12 11:12:39 2019][2937681.123793] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Apr 12 11:12:39 2019][2937681.130856] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Fri Apr 12 11:12:39 2019][2937681.137581] [] ? lnet_ni_send+0x44/0xd0 [lnet] [Fri Apr 12 11:12:39 2019][2937681.143880] [] ? lnet_send+0x82/0x1c0 [lnet] [Fri Apr 12 11:12:39 2019][2937681.149998] [] ? LNetPut+0x2cc/0xb50 [lnet] [Fri Apr 12 11:12:39 2019][2937681.156078] [] ? ptl_send_buf+0x146/0x530 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.162739] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:39 2019][2937681.170098] [] ? null_alloc_rs+0x186/0x340 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.176805] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:12:39 2019][2937681.182556] [] ? account_entity_dequeue+0xae/0xd0 [Fri Apr 12 11:12:39 2019][2937681.189094] [] ? dequeue_entity+0x11c/0x5e0 [Fri Apr 12 11:12:39 2019][2937681.195108] [] ? set_next_entity+0x3c/0xe0 [Fri Apr 12 11:12:39 2019][2937681.201038] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:12:39 2019][2937681.206799] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:12:39 2019][2937681.213456] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:12:39 2019][2937681.219642] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937681.225011] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.232064] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:39 2019][2937681.238561] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937681.244354] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.251007] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.258581] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937681.263815] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.270092] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937681.276884] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.283160] mdt_io00_019 S ffff8b4e2960b0c0 0 118639 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937681.290523] Call Trace: [Fri Apr 12 11:12:39 2019][2937681.293156] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937681.298343] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.305218] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937681.311010] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.317491] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.325065] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937681.330126] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.336401] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937681.343020] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.349303] mdt_io03_011 S ffff8b4e29609040 0 118640 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937681.356659] Call Trace: [Fri Apr 12 11:12:39 2019][2937681.359303] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937681.364532] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.371427] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937681.377214] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.383700] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.391272] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937681.396345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.402617] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937681.409240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.415523] mdt_io01_013 S ffff8b5776ff9040 0 118641 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937681.422895] Call Trace: [Fri Apr 12 11:12:39 2019][2937681.425563] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937681.430760] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.437646] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937681.443483] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.449988] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.457572] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937681.462636] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.468914] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937681.475565] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.481858] mdt_io02_017 S ffff8b5776ffa080 0 118642 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937681.489229] Call Trace: [Fri Apr 12 11:12:39 2019][2937681.491860] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937681.497069] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.503951] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937681.509743] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.516233] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.523819] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937681.528876] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.535152] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937681.541772] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.548044] mdt_io00_020 S ffff8b5776ffc100 0 118643 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937681.555445] Call Trace: [Fri Apr 12 11:12:39 2019][2937681.558075] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937681.563291] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.570174] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937681.575964] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.582444] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.590035] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:39 2019][2937681.595117] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.601406] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:39 2019][2937681.608032] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:39 2019][2937681.614323] mdt03_073 S ffff8b5776ffd140 0 118644 2 0x00000080 [Fri Apr 12 11:12:39 2019][2937681.621684] Call Trace: [Fri Apr 12 11:12:39 2019][2937681.624326] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:39 2019][2937681.631279] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.639061] [] schedule+0x29/0x70 [Fri Apr 12 11:12:39 2019][2937681.644253] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.651356] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.658443] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:39 2019][2937681.665589] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:39 2019][2937681.671379] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.678724] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.686756] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:39 2019][2937681.693483] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:39 2019][2937681.700466] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:39 2019][2937681.707220] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.715288] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:39 2019][2937681.722479] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.729640] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:12:39 2019][2937681.736824] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.743488] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.750575] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.757727] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:39 2019][2937681.764461] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:39 2019][2937681.771128] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.778060] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:39 2019][2937681.784886] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.791776] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:39 2019][2937681.799100] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:39 2019][2937681.805682] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.812951] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.820654] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.826970] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.834073] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.841863] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:39 2019][2937681.849138] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.857002] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:39 2019][2937681.863992] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:39 2019][2937681.870440] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:40 2019][2937681.876491] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937681.882982] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937681.890589] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937681.895650] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937681.901927] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937681.908577] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937681.914854] mdt_io02_018 S ffff8b5cca7ae180 0 118645 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937681.922247] Call Trace: [Fri Apr 12 11:12:40 2019][2937681.924889] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937681.930084] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937681.936966] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937681.942758] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937681.949254] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937681.956850] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937681.961909] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937681.968183] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937681.974804] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937681.981074] mdt_io03_012 S ffff8b5cca7aa080 0 118646 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937681.988432] Call Trace: [Fri Apr 12 11:12:40 2019][2937681.991081] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937681.996270] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.003152] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.008943] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.015434] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.023032] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.028094] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.034370] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.040993] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.047275] mdt_io00_021 S ffff8b5cca7a9040 0 118647 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.054646] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.057280] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.062480] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.069364] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.075170] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.081650] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.089225] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.094287] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.100560] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.107179] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.113454] mdt_io01_014 S ffff8b5cf6609040 0 118648 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.120818] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.123466] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.128661] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.135617] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.141424] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.147934] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.155546] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.160615] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.166907] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.173548] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.179819] mdt_io03_013 S ffff8b5cfb63d140 0 118649 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.187180] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.189814] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.195039] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.201931] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.207728] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.214218] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.221792] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.226856] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.233128] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.239963] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.246268] mdt_io00_022 S ffff8b5cfb63e180 0 118650 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.253630] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.256273] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.261467] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.268425] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.274246] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.280785] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.288399] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.293456] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.299733] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.306351] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.312621] mdt_io02_019 S ffff8b5cfb63a080 0 118651 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.319982] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.322614] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.327802] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.334704] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.340523] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.347004] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.354581] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.359637] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.365917] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.372553] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.378826] mdt_io02_020 S ffff8b5cfb63c100 0 118652 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.386208] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.388859] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.394077] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.400983] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.406800] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.413363] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.420968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.426059] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.432337] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.438956] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.445233] mdt_io00_023 S ffff8b778173c100 0 118653 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.452593] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.455227] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.460456] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.467373] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.473161] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.479642] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.487215] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.492275] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.498564] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.505191] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.511471] mdt_io03_014 S ffff8b778173a080 0 118654 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.518835] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.521470] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.526662] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.533553] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.539344] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.545863] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.553467] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.558550] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.564830] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.571447] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.577722] mdt_io02_021 S ffff8b7781738000 0 118655 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.585092] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.587723] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.592931] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.599816] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.605606] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.612088] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.619660] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.624722] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.630996] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.637619] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.643892] mdt_io00_024 S ffff8b778173d140 0 118656 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.651257] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.653903] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.659091] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.665967] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.671760] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.678244] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.685815] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.690893] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.697169] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.703818] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.710091] mdt_io02_022 S ffff8b7781739040 0 118657 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.717505] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.720145] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.725345] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.732308] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.738185] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.744759] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.752333] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.757397] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.763683] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.770454] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.776767] mdt03_074 S ffff8b889f7e8000 0 118658 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.784165] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.786805] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.791996] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.798874] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:40 2019][2937682.804667] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.811148] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:40 2019][2937682.818726] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:40 2019][2937682.823809] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.830094] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:40 2019][2937682.836717] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:40 2019][2937682.842988] mdt_io03_015 D ffff8b867b7d2080 0 118659 2 0x00000080 [Fri Apr 12 11:12:40 2019][2937682.850372] Call Trace: [Fri Apr 12 11:12:40 2019][2937682.853019] [] schedule+0x29/0x70 [Fri Apr 12 11:12:40 2019][2937682.858179] [] osd_do_bio.isra.35+0x785/0xb40 [osd_ldiskfs] [Fri Apr 12 11:12:40 2019][2937682.865626] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:12:40 2019][2937682.871055] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Apr 12 11:12:41 2019][2937682.879033] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Apr 12 11:12:41 2019][2937682.886008] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Apr 12 11:12:41 2019][2937682.893173] [] ? kiblnd_send+0x357/0xa20 [ko2iblnd] [Fri Apr 12 11:12:41 2019][2937682.899975] [] ? lnet_ni_send+0x44/0xd0 [lnet] [Fri Apr 12 11:12:41 2019][2937682.906347] [] ? lnet_send+0x82/0x1c0 [lnet] [Fri Apr 12 11:12:41 2019][2937682.912554] [] ? LNetPut+0x2cc/0xb50 [lnet] [Fri Apr 12 11:12:41 2019][2937682.918802] [] ? ptl_send_buf+0x146/0x530 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937682.925553] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:41 2019][2937682.932916] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937682.940028] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937682.946654] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:12:41 2019][2937682.952437] [] ? account_entity_dequeue+0xae/0xd0 [Fri Apr 12 11:12:41 2019][2937682.958981] [] ? dequeue_entity+0x11c/0x5e0 [Fri Apr 12 11:12:41 2019][2937682.964991] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:12:41 2019][2937682.971279] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:12:41 2019][2937682.977207] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:12:41 2019][2937682.982958] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:12:41 2019][2937682.989598] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:12:41 2019][2937682.995869] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.001236] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.008293] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.014086] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.020741] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.028322] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:41 2019][2937683.033784] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.040344] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:41 2019][2937683.047137] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.053409] mdt03_075 S ffff8b867b7d30c0 0 118660 2 0x00000080 [Fri Apr 12 11:12:41 2019][2937683.060765] Call Trace: [Fri Apr 12 11:12:41 2019][2937683.063397] [] schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.068588] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.075469] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.081262] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.087745] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.095322] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:41 2019][2937683.100392] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.106680] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:41 2019][2937683.113319] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.119902] mdt_io00_025 S ffff8b867b7d1040 0 118661 2 0x00000080 [Fri Apr 12 11:12:41 2019][2937683.127311] Call Trace: [Fri Apr 12 11:12:41 2019][2937683.129945] [] schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.135157] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.142043] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.147851] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.154332] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.161912] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:41 2019][2937683.166977] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.173249] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:41 2019][2937683.179873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.186144] mdt03_076 S ffff8b867b7d5140 0 118662 2 0x00000080 [Fri Apr 12 11:12:41 2019][2937683.193543] Call Trace: [Fri Apr 12 11:12:41 2019][2937683.196174] [] schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.201378] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.208293] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.214097] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.220615] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.228210] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:41 2019][2937683.233277] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.239554] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:41 2019][2937683.246182] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.252464] mdt_io02_023 S ffff8b867b7d0000 0 118663 2 0x00000080 [Fri Apr 12 11:12:41 2019][2937683.259835] Call Trace: [Fri Apr 12 11:12:41 2019][2937683.262468] [] schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.267672] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.274585] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.280396] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.289214] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.296821] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:41 2019][2937683.301880] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.308160] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:41 2019][2937683.314790] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.321098] mdt03_077 S ffff8b867b7d6180 0 118664 2 0x00000080 [Fri Apr 12 11:12:41 2019][2937683.328496] Call Trace: [Fri Apr 12 11:12:41 2019][2937683.331164] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:41 2019][2937683.338103] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.345852] [] schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.351034] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.358118] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.365205] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:41 2019][2937683.372345] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.378131] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.385472] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.393524] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.401294] [] mdt_dom_discard_data+0x101/0x130 [mdt] [Fri Apr 12 11:12:41 2019][2937683.408213] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.415998] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.424021] [] mdt_reint_unlink+0x331/0x1480 [mdt] [Fri Apr 12 11:12:41 2019][2937683.430652] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:41 2019][2937683.436854] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:41 2019][2937683.443571] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:41 2019][2937683.450549] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:41 2019][2937683.456458] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.463558] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.471312] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:41 2019][2937683.478583] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.486455] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.493422] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:41 2019][2937683.499871] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:41 2019][2937683.505929] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.512411] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.519987] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:41 2019][2937683.525047] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.531324] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:41 2019][2937683.537945] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.544217] mdt_io03_016 S ffff8b8c307ab0c0 0 118665 2 0x00000080 [Fri Apr 12 11:12:41 2019][2937683.551588] Call Trace: [Fri Apr 12 11:12:41 2019][2937683.554221] [] schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.559412] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.566294] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.572084] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.578574] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.586149] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:41 2019][2937683.591208] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.597486] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:41 2019][2937683.604104] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.610379] mdt_io01_015 S ffff8b8c307aa080 0 118666 2 0x00000080 [Fri Apr 12 11:12:41 2019][2937683.617742] Call Trace: [Fri Apr 12 11:12:41 2019][2937683.620373] [] schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.625562] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.632447] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.638234] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.644726] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.652298] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:41 2019][2937683.657354] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.663630] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:41 2019][2937683.670250] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:41 2019][2937683.676523] mdt03_078 S ffff8b6cf7ef0000 0 118667 2 0x00000080 [Fri Apr 12 11:12:41 2019][2937683.683886] Call Trace: [Fri Apr 12 11:12:41 2019][2937683.686526] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:41 2019][2937683.693449] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.701200] [] schedule+0x29/0x70 [Fri Apr 12 11:12:41 2019][2937683.706385] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.713475] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.720556] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:41 2019][2937683.727696] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:41 2019][2937683.733490] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.740848] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.748873] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:41 2019][2937683.755598] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:41 2019][2937683.762575] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:41 2019][2937683.769317] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.777335] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:41 2019][2937683.784527] [] ? req_capsule_get_size+0x31/0x70 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.791679] [] mdt_getattr_name_lock+0x101d/0x1c30 [mdt] [Fri Apr 12 11:12:41 2019][2937683.798867] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.805529] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.812628] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.819786] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:41 2019][2937683.826509] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:41 2019][2937683.833161] [] ? ldlm_lock_create+0xa4/0xa40 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.840053] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:41 2019][2937683.846878] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:41 2019][2937683.853773] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:42 2019][2937683.861096] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:42 2019][2937683.867715] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.874984] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.882733] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.889049] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.896150] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.903909] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:42 2019][2937683.911179] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.919057] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.926028] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:42 2019][2937683.932479] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:42 2019][2937683.938531] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.945020] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.952590] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937683.957653] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937683.963930] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937683.970551] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937683.976824] mdt_io03_017 S ffff8b8b5efd8000 0 118668 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937683.984184] Call Trace: [Fri Apr 12 11:12:42 2019][2937683.986817] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937683.992008] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937683.998892] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.004701] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.011188] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.018765] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.023823] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.030101] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.036719] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.042994] mdt03_079 S ffff8b8b5efd9040 0 118669 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.050355] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.052991] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.058178] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.065062] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.070852] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.077346] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.084918] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.089977] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.096257] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.102874] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.109150] mdt_io02_024 S ffff8b8b5efda080 0 118670 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.116510] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.119146] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.124336] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.131216] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.137005] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.143497] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.151071] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.156131] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.162407] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.169028] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.175302] mdt_io00_026 S ffff8b8b5efdb0c0 0 118671 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.182664] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.185297] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.190487] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.197370] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.203162] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.209648] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.217226] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.222286] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.228564] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.235183] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.241457] mdt_io02_025 S ffff8b8b5efdc100 0 118672 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.248829] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.251458] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.256652] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.263533] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.269322] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.275820] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.283398] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.288457] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.294737] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.301355] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.307628] mdt_io00_027 S ffff8b8b5efdd140 0 118673 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.314998] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.317631] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.322825] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.329709] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.335514] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.341999] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.349577] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.354638] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.360915] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.367533] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.373809] mdt_io00_028 S ffff8b8b5efde180 0 118674 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.381170] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.383803] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.388993] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.395877] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.401673] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.408169] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.415743] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.420801] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.427077] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.433695] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.439972] mdt_io03_018 S ffff8b8a7cbb0000 0 118675 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.447333] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.449965] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.455158] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.462038] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.467832] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.474315] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.481887] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.486945] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.493223] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.499843] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.506116] mdt_out01_005 S ffff8b8a7cbb1040 0 118676 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.513480] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.516111] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.521309] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.528195] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.533989] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.540489] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.548066] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.553126] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.559402] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.566023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.572296] mdt_io02_026 S ffff8b8a7cbb2080 0 118677 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.579669] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.582300] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.587493] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.594374] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.600168] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.606661] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.614240] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.619298] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.625575] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.632195] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.638468] mdt03_080 S ffff8b8a7cbb30c0 0 118678 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.645831] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.648463] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.653657] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.660536] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.666329] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.672819] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.680392] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.685452] [] ? insert_kthread_work+0x40/0nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.698348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.704624] mdt_io03_019 S ffff8b8a7cbb4100 0 118679 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.712003] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.714635] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.719824] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.726708] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.732498] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.738988] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.746563] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.751625] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.757901] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.764522] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.770794] mdt_io02_027 S ffff8b8a7cbb5140 0 118680 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.778155] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.780789] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.785978] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.792862] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.798654] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.805143] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.812719] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:42 2019][2937684.817776] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.824053] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:42 2019][2937684.830675] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:42 2019][2937684.836947] mdt_io02_028 S ffff8b8a7cbb6180 0 118681 2 0x00000080 [Fri Apr 12 11:12:42 2019][2937684.844311] Call Trace: [Fri Apr 12 11:12:42 2019][2937684.846941] [] schedule+0x29/0x70 [Fri Apr 12 11:12:42 2019][2937684.852143] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:42 2019][2937684.859024] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:42 2019][2937684.864818] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937684.871304] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937684.878890] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937684.883966] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937684.890242] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937684.896863] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937684.903138] mdt03_081 S ffff8b8a49680000 0 118682 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937684.910500] Call Trace: [Fri Apr 12 11:12:43 2019][2937684.913133] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937684.918323] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937684.925204] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937684.930996] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937684.937488] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937684.945062] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937684.950120] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937684.956397] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937684.963017] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937684.969290] mdt_io03_020 S ffff8b8a49681040 0 118683 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937684.976653] Call Trace: [Fri Apr 12 11:12:43 2019][2937684.979286] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937684.984474] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937684.991358] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937684.997150] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.003636] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.011216] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.016276] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.022552] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.029171] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.035443] mdt_io02_029 S ffff8b8a49682080 0 118684 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.042807] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.045440] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.050631] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.057515] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.063306] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.069801] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.077378] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.082440] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.088715] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.095335] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.101608] mdt_io00_029 S ffff8b8a496830c0 0 118685 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.108980] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.111610] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.116800] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.123685] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.129483] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.135973] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.143551] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.148610] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.154885] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.161506] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.167781] mdt03_082 S ffff8b8a49684100 0 118686 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.175150] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.177785] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.182974] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.189855] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.195652] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.202149] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.209723] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.214780] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.221057] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.227676] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.233951] mdt_io03_021 S ffff8b8a49685140 0 118687 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.241338] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.243973] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.249163] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.256046] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.261839] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.268325] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.275902] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.280962] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.287238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.293858] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.300131] mdt_io01_016 S ffff8b8a49686180 0 118688 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.307493] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.310127] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.315326] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.322211] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.328017] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.334508] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.342082] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.347142] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.353419] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.360038] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.366311] mdt_io02_030 S ffff8b8bc6348000 0 118689 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.373673] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.376309] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.381496] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.388382] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.394175] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.400670] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.408244] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.413303] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.419581] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.426200] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.432474] mdt03_083 S ffff8b8bc6349040 0 118690 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.439844] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.442479] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.447679] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.454564] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.460368] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.466859] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.474437] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.479505] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.485780] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.492399] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.498682] mdt_io00_030 S ffff8b8bc634a080 0 118691 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.506042] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.508679] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.513882] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.520771] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.526570] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.533068] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.540684] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.545742] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.552048] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.558666] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.564938] mdt_io02_031 S ffff8b8bc634b0c0 0 118692 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.572302] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.574933] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.580126] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.587141] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.593002] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.599489] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.607062] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.612155] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.618445] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.625177] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.631496] mdt_io03_022 S ffff8b8bc634c100 0 118693 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.638864] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.641516] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.646730] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.654221] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.660256] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.666745] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.674316] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:43 2019][2937685.679376] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.685651] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:43 2019][2937685.692271] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:43 2019][2937685.698546] mdt03_084 S ffff8b8bc634d140 0 118694 2 0x00000080 [Fri Apr 12 11:12:43 2019][2937685.705912] Call Trace: [Fri Apr 12 11:12:43 2019][2937685.708588] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.716331] [] schedule+0x29/0x70 [Fri Apr 12 11:12:43 2019][2937685.721514] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.728686] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.735750] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:43 2019][2937685.741560] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.749671] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:43 2019][2937685.757034] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.764116] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.772171] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:43 2019][2937685.778839] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:43 2019][2937685.785553] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:43 2019][2937685.791948] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:43 2019][2937685.799009] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:43 2019][2937685.806701] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:43 2019][2937685.814046] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:43 2019][2937685.820334] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:43 2019][2937685.827796] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:43 2019][2937685.835101] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:43 2019][2937685.841666] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:43 2019][2937685.848822] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:43 2019][2937685.855804] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:43 2019][2937685.862451] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:43 2019][2937685.870132] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:44 2019][2937685.876419] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:44 2019][2937685.882355] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:44 2019][2937685.888903] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:44 2019][2937685.895096] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:44 2019][2937685.901828] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:44 2019][2937685.908850] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:44 2019][2937685.914750] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937685.921849] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937685.929603] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:44 2019][2937685.936863] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937685.944733] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937685.951697] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:44 2019][2937685.958145] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:44 2019][2937685.964202] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937685.970689] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937685.978263] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937685.983322] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937685.989600] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937685.996220] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.002491] mdt_io00_031 S ffff8b8bc634e180 0 118695 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.009849] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.012479] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.017668] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.024544] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.030337] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.036815] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.044389] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.049451] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.055727] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.062347] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.068638] mdt_io02_032 S ffff8b8cf1ff8000 0 118696 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.075993] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.078626] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.083826] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.090710] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.096507] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.102992] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.110563] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.115624] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.121899] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.128519] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.134794] mdt_io03_023 S ffff8b6cf96530c0 0 118697 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.142158] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.144793] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.149991] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.156876] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.162679] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.169161] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.176736] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.181796] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.188071] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.194693] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.200964] mdt_io01_017 S ffff8b5cfab54100 0 118698 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.208328] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.210962] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.216151] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.223033] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.228870] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.235383] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.242959] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.248016] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.254298] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.260915] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.267190] mdt03_085 S ffff8b750a6c30c0 0 118699 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.274589] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.277229] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.282419] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.289318] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.295113] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.301656] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.309245] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.314321] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.320595] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.327250] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.333536] mdt_io02_033 S ffff8b750a6c4100 0 118700 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.340904] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.343537] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.348732] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.355612] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:12:44 2019][2937686.362592] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.368413] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.374970] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.382654] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.387713] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.394001] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.400627] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.407413] mdt_io02_034 S ffff8b750a6c2080 0 118701 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.415017] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.417661] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.422878] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.430019] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.435884] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.442387] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.449976] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.455037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.461500] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.468153] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.474421] mdt_io01_018 S ffff8b750a6c0000 0 118702 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.481786] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.484419] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.489663] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.496564] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.502413] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.508900] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.516507] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.521568] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.527836] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.534511] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.540812] mdt_io00_032 S ffff8b750a6c1040 0 118703 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.548193] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.550834] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.556025] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.562907] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.568711] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.575411] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.582997] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:44 2019][2937686.588057] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.594330] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:44 2019][2937686.600951] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:44 2019][2937686.607227] mdt03_086 S ffff8b8c44548000 0 118704 2 0x00000080 [Fri Apr 12 11:12:44 2019][2937686.614589] Call Trace: [Fri Apr 12 11:12:44 2019][2937686.617264] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.625013] [] schedule+0x29/0x70 [Fri Apr 12 11:12:44 2019][2937686.630195] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.637282] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.644338] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:44 2019][2937686.650127] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.657406] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:44 2019][2937686.664766] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.671597] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.679642] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:44 2019][2937686.686371] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:44 2019][2937686.694345] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:44 2019][2937686.701157] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:44 2019][2937686.708296] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:44 2019][2937686.716020] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:44 2019][2937686.723359] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:44 2019][2937686.729677] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:44 2019][2937686.737097] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:44 2019][2937686.744104] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:44 2019][2937686.750528] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.757704] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:44 2019][2937686.764875] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:44 2019][2937686.771545] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:44 2019][2937686.779295] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:44 2019][2937686.785616] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:44 2019][2937686.791598] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:44 2019][2937686.798158] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:44 2019][2937686.804359] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:44 2019][2937686.811084] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:44 2019][2937686.818086] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:44 2019][2937686.823991] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.831092] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.838850] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:44 2019][2937686.846208] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:44 2019][2937686.854110] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937686.861084] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:45 2019][2937686.867534] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:45 2019][2937686.873619] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937686.880233] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937686.887861] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937686.892923] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937686.899235] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937686.905887] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937686.913016] mdt_io03_024 S ffff8b8c44549040 0 118705 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937686.920572] Call Trace: [Fri Apr 12 11:12:45 2019][2937686.923211] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937686.928433] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937686.935411] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937686.943825] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937686.950347] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937686.957952] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937686.963025] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937686.969325] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937686.975971] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937686.982340] mdt_io02_035 S ffff8b8c4454a080 0 118706 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937686.989762] Call Trace: [Fri Apr 12 11:12:45 2019][2937686.992414] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937686.997642] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.004548] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.010388] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.016895] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.024468] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.029527] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.035801] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.042423] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.048696] mdt_io02_036 S ffff8b8c4454b0c0 0 118707 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.056060] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.058689] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.063880] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.070765] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.076555] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.083037] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.090610] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.095672] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:12:45 2019][2937687.101945] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.108223] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.114843] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.121116] mdt_io03_025 S ffff8b8c4454c100 0 118708 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.128473] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.131101] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.136288] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.143168] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.148957] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.155442] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.163019] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.168095] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.174389] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.181011] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.187431] mdt03_087 S ffff8b8c4454d140 0 118709 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.194840] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.197477] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.202635] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:12:45 2019][2937687.208680] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:45 2019][2937687.216077] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:12:45 2019][2937687.222621] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [Fri Apr 12 11:12:45 2019][2937687.229555] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.237343] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.244444] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.251496] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.257285] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.264540] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:45 2019][2937687.271924] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.278759] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.286917] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:45 2019][2937687.293567] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:45 2019][2937687.300281] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:45 2019][2937687.306683] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:45 2019][2937687.313793] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:45 2019][2937687.321500] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:45 2019][2937687.328820] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:45 2019][2937687.335110] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:45 2019][2937687.342524] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:45 2019][2937687.349416] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:45 2019][2937687.355823] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.362976] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:45 2019][2937687.369957] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:45 2019][2937687.376630] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:45 2019][2937687.384355] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:45 2019][2937687.390649] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:45 2019][2937687.396595] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:45 2019][2937687.403154] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:45 2019][2937687.409371] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:45 2019][2937687.416089] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:45 2019][2937687.423065] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:45 2019][2937687.428976] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.436086] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.443847] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:45 2019][2937687.451114] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.458996] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.465987] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:45 2019][2937687.472441] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:45 2019][2937687.478495] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.484984] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.492566] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.497633] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.503910] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.510530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.516811] mdt_io03_026 S ffff8b8c4454e180 0 118710 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.524184] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.526819] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.532012] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.538900] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.544695] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.551180] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.558754] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.563833] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.570125] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.576745] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.583019] mdt_io02_037 S ffff8b8c04c20000 0 118711 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.590390] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.593041] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.598252] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.605158] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.610977] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.617465] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.625065] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.630122] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.636397] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.643019] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.649310] mdt03_088 S ffff8b8c04c21040 0 118712 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.656787] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.659420] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.664612] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.671503] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.677293] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.683770] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.691347] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.696408] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.702683] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.709306] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.715587] mdt_io02_038 S ffff8b8c04c22080 0 118713 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.722950] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.725582] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.730775] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.737654] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.743450] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.749939] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.757516] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.762581] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.768857] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.775475] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.781754] mdt_io01_019 S ffff8b8c04c230c0 0 118714 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.789113] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.791747] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.796935] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.803844] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:45 2019][2937687.809656] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.816144] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:45 2019][2937687.823718] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:45 2019][2937687.828780] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.835061] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:45 2019][2937687.841683] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:45 2019][2937687.847964] mdt_io02_039 S ffff8b8c04c24100 0 118715 2 0x00000080 [Fri Apr 12 11:12:45 2019][2937687.855345] Call Trace: [Fri Apr 12 11:12:45 2019][2937687.857980] [] schedule+0x29/0x70 [Fri Apr 12 11:12:45 2019][2937687.863177] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937687.870067] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937687.875868] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937687.882351] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937687.889922] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937687.894981] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937687.901259] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937687.907880] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937687.914170] mdt03_089 S ffff8b8c04c25140 0 118716 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937687.921535] Call Trace: [Fri Apr 12 11:12:46 2019][2937687.924200] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937687.929400] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937687.936283] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937687.942085] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937687.948572] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937687.956150] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937687.961208] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937687.967484] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937687.974104] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937687.980403] mdt_io03_027 S ffff8b8c04c26180 0 118717 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937687.987768] Call Trace: [Fri Apr 12 11:12:46 2019][2937687.990408] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937687.995598] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.002489] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.008290] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.014778] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.022352] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.027413] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.033688] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.040310] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.046583] mdt_io02_040 S ffff8b8ce43c8000 0 118718 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.053946] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.056578] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.061770] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.068660] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.074649] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.081163] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.088742] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.093802] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.100078] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.106695] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.112989] mdt03_090 S ffff8b8ce43c9040 0 118719 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.120353] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.122994] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.128185] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.135067] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.140859] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.147360] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.154947] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.160007] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.166283] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.172904] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.179179] mdt_io03_028 S ffff8b8ce43ca080 0 118720 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.186542] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.189182] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.194386] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.201290] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.207080] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.213560] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.221147] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.226223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.232535] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.239683] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.246316] mdt03_091 S ffff8b8ce43cb0c0 0 118721 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.253692] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.256345] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.261533] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.268413] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.274205] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.280694] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.288282] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.293349] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.299623] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.306268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.312584] mdt_io01_020 S ffff8b8ce43cc100 0 118722 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.319953] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.322597] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.327802] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.334723] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.340533] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.347018] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.354616] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.359675] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.365948] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.372570] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.378843] mdt_io03_029 S ffff8b8ce43cd140 0 118723 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.386208] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.388839] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.394033] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.400912] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.406703] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.413184] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.420757] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.425821] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.432095] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.438713] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.444980] mdt03_092 S ffff8b8ce43ce180 0 118724 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.452334] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.455000] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.462749] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.467925] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.475002] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.482059] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.487841] [] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.495106] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [Fri Apr 12 11:12:46 2019][2937688.502455] [] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.509532] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.517935] [] ? mdt_init_export+0x270/0x270 [mdt] [Fri Apr 12 11:12:46 2019][2937688.524648] [] osp_md_object_lock+0x162/0x2d0 [osp] [Fri Apr 12 11:12:46 2019][2937688.531379] [] lod_object_lock+0xf3/0x7b0 [lod] [Fri Apr 12 11:12:46 2019][2937688.537789] [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] [Fri Apr 12 11:12:46 2019][2937688.544869] [] ? cfs_hash_bd_lookup_intent+0x63/0x170 [libcfs] [Fri Apr 12 11:12:46 2019][2937688.552657] [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Fri Apr 12 11:12:46 2019][2937688.560070] [] mdd_object_lock+0x3e/0xe0 [mdd] [Fri Apr 12 11:12:46 2019][2937688.566389] [] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] [Fri Apr 12 11:12:46 2019][2937688.573811] [] mdt_remote_object_lock+0x2a/0x30 [mdt] [Fri Apr 12 11:12:46 2019][2937688.580704] [] mdt_rename_lock+0xbe/0x4b0 [mdt] [Fri Apr 12 11:12:46 2019][2937688.589532] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.596734] [] ? mdt_version_get_check+0x32/0xb0 [mdt] [Fri Apr 12 11:12:46 2019][2937688.603747] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:46 2019][2937688.610436] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:46 2019][2937688.618165] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:46 2019][2937688.625011] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:46 2019][2937688.630971] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:46 2019][2937688.637557] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:46 2019][2937688.643754] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:46 2019][2937688.650479] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:46 2019][2937688.657496] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:46 2019][2937688.663417] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.673948] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.681709] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:46 2019][2937688.689000] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.696886] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.703859] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:46 2019][2937688.710314] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:46 2019][2937688.716372] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.722859] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.730433] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.735497] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.741800] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.748420] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.754692] mdt_io03_030 S ffff8b8575fd8000 0 118725 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.762053] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.764686] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.769902] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.776798] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.782603] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.789117] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.796709] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:46 2019][2937688.801771] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.808045] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:46 2019][2937688.814687] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:46 2019][2937688.820958] mdt_io00_033 S ffff8b8575fd9040 0 118726 2 0x00000080 [Fri Apr 12 11:12:46 2019][2937688.828330] Call Trace: [Fri Apr 12 11:12:46 2019][2937688.830971] [] schedule+0x29/0x70 [Fri Apr 12 11:12:46 2019][2937688.836175] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.843051] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:46 2019][2937688.848849] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:46 2019][2937688.855335] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937688.862906] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937688.867967] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937688.874244] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937688.880864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937688.887137] mdt_io03_031 S ffff8b8575fda080 0 118727 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937688.894502] Call Trace: [Fri Apr 12 11:12:47 2019][2937688.897136] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937688.906345] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937688.913427] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937688.919241] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937688.925731] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937688.933311] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937688.938384] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937688.944675] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937688.951299] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937688.957619] mdt03_093 S ffff8b8575fdb0c0 0 118728 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937688.965008] Call Trace: [Fri Apr 12 11:12:47 2019][2937688.967640] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937688.972836] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937688.979718] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937688.985514] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937688.992008] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937688.999606] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.004697] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.010977] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.017595] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.023894] mdt_io00_034 S ffff8b8575fdc100 0 118729 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.032126] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.034778] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.040034] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.046919] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.052735] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.059234] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.066822] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.071896] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.078198] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.084838] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.091133] mdt_io02_041 S ffff8b8575fdd140 0 118730 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.098511] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.101145] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.106357] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.113265] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.123165] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.131239] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.139172] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.144396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.150680] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.157342] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.163640] mdt_io01_021 S ffff8b8575fde180 0 118731 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.171055] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.173701] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.178910] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.185805] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.191649] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.198149] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.205728] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.210786] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.217063] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.223686] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.229956] mdt_io03_032 S ffff8b8c476a8000 0 118732 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.237327] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.239961] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.245157] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.252061] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.257865] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.264385] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.271963] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.277026] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.283303] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.289922] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.296214] mdt_io02_042 S ffff8b8c476a9040 0 118733 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.303830] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.306490] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.311682] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.318561] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.324381] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.330888] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.338460] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.343530] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:12:47 2019][2937689.349823] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.356132] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.362764] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.369051] mdt_io00_035 S ffff8b8c476aa080 0 118734 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.376451] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.379128] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.388516] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.395760] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.401559] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.408157] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.415771] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.421483] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.428020] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.434681] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.440960] mdt03_094 S ffff8b8c476ab0c0 0 118735 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.448323] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.450976] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.456169] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.463147] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.468980] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.475515] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.483112] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.488631] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.495028] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.501647] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.507920] mdt_io03_033 S ffff8b8c476ac100 0 118736 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.515303] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.518013] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.523217] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.530147] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.536021] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.545522] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.553306] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.558419] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.564722] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.571382] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.577673] mdt_io01_022 S ffff8b8c476ad140 0 118737 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.585044] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.587677] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.592874] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.599791] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.605600] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.612108] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.619780] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.624844] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.631133] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.637790] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.644568] mdt03_095 S ffff8b8c476ae180 0 118738 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.652366] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.655004] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.660207] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.667100] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.672893] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.679382] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.686976] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.692044] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.698337] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.704957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.711227] mdt03_096 S ffff8b8cfe506180 0 118739 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.718591] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.721223] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.726417] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.733305] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.739099] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.745588] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.753171] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.758227] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.764526] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.771156] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.777426] mdt_io03_034 S ffff8b8be5eb8000 0 118740 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.784807] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.787467] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.792672] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.799603] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:47 2019][2937689.805468] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.811964] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:47 2019][2937689.819564] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:47 2019][2937689.824628] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.830912] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:47 2019][2937689.837535] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:47 2019][2937689.844076] mdt_io02_043 S ffff8b8be5eb9040 0 118741 2 0x00000080 [Fri Apr 12 11:12:47 2019][2937689.851461] Call Trace: [Fri Apr 12 11:12:47 2019][2937689.854095] [] schedule+0x29/0x70 [Fri Apr 12 11:12:47 2019][2937689.859287] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937689.866196] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937689.872021] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937689.878512] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937689.886096] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937689.891164] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937689.897456] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937689.904093] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937689.910496] mdt_io02_044 S ffff8b8be5eba080 0 118742 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937689.917897] Call Trace: [Fri Apr 12 11:12:48 2019][2937689.920538] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937689.925739] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937689.932628] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937689.941178] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937689.947756] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937689.955843] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937689.961159] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937689.967451] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937689.974085] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937689.980781] mdt_io00_036 S ffff8b8be5ebb0c0 0 118743 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937689.989090] Call Trace: [Fri Apr 12 11:12:48 2019][2937689.991816] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937689.997042] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.003930] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.009727] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.016213] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.023801] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.028859] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.035136] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.041755] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.048031] mdt_io02_045 S ffff8b8be5ebc100 0 118744 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.055393] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.058026] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.063223] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.069876] LustreError: dumping log to /tmp/lustre-log.1555092768.118271 [Fri Apr 12 11:12:48 2019][2937690.077094] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.082899] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.089387] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.096968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.102027] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.108304] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.114950] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.121230] mdt_io03_035 S ffff8b869a65d140 0 118745 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.128612] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.131256] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.136504] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.143458] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.149261] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.155747] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.163319] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.168383] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.174705] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.181340] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.187611] mdt_io01_023 S ffff8b8be5ebd140 0 118746 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.194992] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.197656] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.202878] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.209778] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.215569] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.222061] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.229648] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.234724] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.241027] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.247668] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.253937] mdt_io00_037 S ffff8b8be5ebe180 0 118747 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.261302] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.263942] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.269150] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.276032] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.281823] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.288307] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.295879] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.300940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.307238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.313896] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.320172] mdt_io02_046 S ffff8b8c61258000 0 118748 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.327542] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.330189] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.337597] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.344492] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.350294] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.356802] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.364389] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.369452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.375730] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.382378] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.388652] mdt03_097 S ffff8b869a65e180 0 118749 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.396035] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.398676] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:48 2019][2937690.405561] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:48 2019][2937690.411701] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.419459] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.424650] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.431739] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.438846] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:48 2019][2937690.446000] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.455439] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.462988] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.471038] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.478798] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:48 2019][2937690.487906] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.495757] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.503777] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:48 2019][2937690.510433] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:48 2019][2937690.518121] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:48 2019][2937690.524410] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:48 2019][2937690.530345] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:48 2019][2937690.536892] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:48 2019][2937690.543094] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:48 2019][2937690.549812] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:48 2019][2937690.556788] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:48 2019][2937690.562690] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.569806] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.577581] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:48 2019][2937690.584846] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.592721] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.599688] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:48 2019][2937690.606613] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:48 2019][2937690.612788] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.619306] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.626886] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.631945] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.638256] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.645166] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.651540] mdt_io01_024 S ffff8b8aeca00000 0 118750 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.658921] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.661566] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.666761] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.673642] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:12:48 2019][2937690.680621] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.686456] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.692964] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.700558] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.705625] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.711898] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.718522] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.724812] mdt_io00_038 S ffff8b8af6250000 0 118751 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.732208] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.734850] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.740038] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.746921] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.752722] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.759201] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.766848] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.771933] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.778207] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.784827] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.791101] mdt_io02_047 S ffff8b8af6251040 0 118752 2 0x00000080 [Fri Apr 12 11:12:48 2019][2937690.798464] Call Trace: [Fri Apr 12 11:12:48 2019][2937690.801098] [] schedule+0x29/0x70 [Fri Apr 12 11:12:48 2019][2937690.806302] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.813188] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:48 2019][2937690.819093] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.825612] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:48 2019][2937690.833211] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:48 2019][2937690.838277] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.844552] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:48 2019][2937690.851171] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:48 2019][2937690.857443] mdt03_098 S ffff8b8af6252080 0 118753 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937690.864799] Call Trace: [Fri Apr 12 11:12:49 2019][2937690.867432] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937690.872621] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937690.879507] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937690.885295] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937690.891780] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937690.899486] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937690.904548] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937690.910822] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937690.917439] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937690.923714] mdt_io01_025 S ffff8b8af62530c0 0 118754 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937690.931094] Call Trace: [Fri Apr 12 11:12:49 2019][2937690.933728] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937690.938928] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937690.945828] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937690.951616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937690.958099] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937690.965670] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937690.970729] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937690.976997] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937690.983630] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937690.989928] mdt03_099 S ffff8b5cfab530c0 0 118755 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937690.997439] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.000099] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.005310] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.012235] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.018039] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.024534] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.032124] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.037192] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.043464] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.050085] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.056360] mdt_io03_036 S ffff8b5cfab55140 0 118756 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.063723] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.066361] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.071568] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.078466] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.084256] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.090749] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.098347] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.103409] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.109886] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.116511] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.122780] mdt_io02_048 S ffff8b5cfab56180 0 118757 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.130288] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.132944] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.138155] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.145042] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.150858] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.157358] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.164932] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.170023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.176303] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.182979] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.189319] mdt_io02_049 S ffff8b8a7cba0000 0 118758 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.196724] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.199395] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.204625] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.211512] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.217360] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.223846] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.231427] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.236557] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.242880] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.249512] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.255784] mdt_io01_026 S ffff8b8a7cba1040 0 118759 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.263148] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.265778] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.270971] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.277852] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.283644] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.290127] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.297701] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.302763] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.309055] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.315764] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.322052] mdt_io02_050 S ffff8b8a7cba2080 0 118760 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.329423] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.332179] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.337414] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.344305] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.352705] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.359215] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.366794] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.371862] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.378137] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.384757] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.391029] mdt_io03_037 S ffff8b8a7cba30c0 0 118761 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.398394] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.401025] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.406214] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.413098] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.418914] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.425397] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.432970] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.438029] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.444298] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.450919] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.457193] mdt03_100 S ffff8b8a7cba4100 0 118762 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.464557] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.468172] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:49 2019][2937691.475707] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:49 2019][2937691.482047] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.489823] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.495020] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.503295] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.510402] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:49 2019][2937691.517548] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.523351] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.530699] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.538742] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.546536] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:49 2019][2937691.553027] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.560808] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.568829] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:49 2019][2937691.575588] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:49 2019][2937691.583314] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:49 2019][2937691.589709] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:49 2019][2937691.595649] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:49 2019][2937691.602207] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:49 2019][2937691.608410] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:49 2019][2937691.615173] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:49 2019][2937691.622180] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:49 2019][2937691.628128] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.635232] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.642982] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:49 2019][2937691.650264] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.658554] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.665569] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:49 2019][2937691.672021] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:49 2019][2937691.678079] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.684568] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.692146] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.697206] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.703481] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.710102] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.716373] mdt_io02_051 S ffff8b8a7cba5140 0 118763 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.723748] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.726400] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.731591] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.738472] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.744310] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.750835] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.758422] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:49 2019][2937691.763511] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:12:49 2019][2937691.769779] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.776051] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:49 2019][2937691.782671] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:49 2019][2937691.788944] mdt03_101 S ffff8b8a7cba6180 0 118764 2 0x00000080 [Fri Apr 12 11:12:49 2019][2937691.796359] Call Trace: [Fri Apr 12 11:12:49 2019][2937691.798999] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:49 2019][2937691.805883] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:12:49 2019][2937691.812025] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.819773] [] schedule+0x29/0x70 [Fri Apr 12 11:12:49 2019][2937691.824956] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.832061] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.839165] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:49 2019][2937691.846332] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:49 2019][2937691.852124] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.859494] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:49 2019][2937691.867941] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937691.875717] [] mdt_rename_lock+0x24b/0x4b0 [mdt] [Fri Apr 12 11:12:50 2019][2937691.882205] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937691.890016] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937691.898106] [] mdt_reint_rename+0x2d5/0x28e0 [mdt] [Fri Apr 12 11:12:50 2019][2937691.904761] [] ? upcall_cache_get_entry+0x218/0x8b0 [obdclass] [Fri Apr 12 11:12:50 2019][2937691.912450] [] ? lu_ucred+0x1e/0x30 [obdclass] [Fri Apr 12 11:12:50 2019][2937691.918737] [] ? mdt_ucred+0x15/0x20 [mdt] [Fri Apr 12 11:12:50 2019][2937691.924698] [] ? mdt_root_squash+0x21/0x430 [mdt] [Fri Apr 12 11:12:50 2019][2937691.931249] [] mdt_reint_rec+0x83/0x210 [mdt] [Fri Apr 12 11:12:50 2019][2937691.937453] [] mdt_reint_internal+0x6e3/0xaf0 [mdt] [Fri Apr 12 11:12:50 2019][2937691.944185] [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] [Fri Apr 12 11:12:50 2019][2937691.951165] [] mdt_reint+0x67/0x140 [mdt] [Fri Apr 12 11:12:50 2019][2937691.957062] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937691.964206] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937691.971990] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:50 2019][2937691.979268] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937691.987174] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937691.994741] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:50 2019][2937692.001502] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:50 2019][2937692.007604] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.014114] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.021697] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.026756] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.033041] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.039659] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.045944] mdt_io01_027 S ffff8b8cf8b00000 0 118765 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.053335] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.056020] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.061233] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.068115] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.073962] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.080486] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.088076] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.093135] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.099420] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.106040] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.112322] mdt_io03_038 S ffff8b8cf8b01040 0 118766 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.119704] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.122344] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.127540] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.134434] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.140235] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.146719] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.154300] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.159357] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.165633] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.172301] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.178572] mdt_io02_052 S ffff8b8cf8b02080 0 118767 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.185952] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.188593] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.193840] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.200721] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.206528] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.213050] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.220637] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.225707] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.231998] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.238632] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.244955] mdt_io00_039 S ffff8b8cf8b030c0 0 118768 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.252315] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.255451] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.261497] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.268412] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.274216] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.280736] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.288315] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.293374] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.299650] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.306268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.312563] mdt_io03_039 S ffff8b8cf8b04100 0 118769 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.319933] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.322567] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.327771] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.334694] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.340486] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.346973] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.354545] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.359604] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.365879] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.372501] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.378775] mdt_io03_040 S ffff8b8cf8b05140 0 118770 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.386141] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.388771] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.393960] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.400855] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.406653] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.413156] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.420744] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.425803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.432078] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.438700] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.444971] mdt03_102 S ffff8b8cf8b06180 0 118771 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.452339] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.454979] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.460169] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.467052] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.472844] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.479331] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.486917] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.491992] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.498277] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.504951] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.511224] mdt_io02_053 S ffff8b8cf9ef0000 0 118772 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.518594] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.521776] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.527291] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.534189] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.540012] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.546529] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.554100] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.559171] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.565457] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.572093] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.578384] mdt_io00_040 S ffff8b8cf9ef1040 0 118773 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.585755] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.588387] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.593589] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.600497] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.606297] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.612785] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.620357] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.625418] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.631696] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.638319] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.644978] mdt_io01_028 S ffff8b8cf9ef2080 0 118774 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.652437] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.655069] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.660320] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.667204] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.672997] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.679485] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.687103] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.692161] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.698454] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.705076] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.711351] mdt_io03_041 S ffff8b8cf9ef30c0 0 118775 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.718758] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.721398] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.726587] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.733470] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.739257] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.745742] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.753345] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.758440] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.764717] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.771366] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.777651] mdt03_103 S ffff8b8cf9ef4100 0 118776 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.785024] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.787656] [] schedule+0x29/0x70 [Fri Apr 12 11:12:50 2019][2937692.792859] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.799739] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:50 2019][2937692.805525] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.812009] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:50 2019][2937692.819582] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:50 2019][2937692.824641] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.830912] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:50 2019][2937692.837529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:50 2019][2937692.843805] mdt03_104 S ffff8b8cf9ef5140 0 118777 2 0x00000080 [Fri Apr 12 11:12:50 2019][2937692.851248] Call Trace: [Fri Apr 12 11:12:50 2019][2937692.853943] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937692.859131] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937692.866013] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937692.871843] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937692.878329] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937692.885914] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937692.890972] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937692.897275] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937692.903938] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937692.910213] mdt_io03_042 S ffff8b8cf9ef6180 0 118778 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937692.917904] Call Trace: [Fri Apr 12 11:12:51 2019][2937692.921054] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937692.926332] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937692.933239] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937692.939056] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937692.945572] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937692.953158] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937692.958229] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937692.964522] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937692.971145] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937692.977414] mdt_io03_043 S ffff8b8cf87c8000 0 118779 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937692.984795] Call Trace: [Fri Apr 12 11:12:51 2019][2937692.987429] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937692.992628] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937692.999520] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.005357] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.011853] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.019424] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.024482] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.030764] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.037383] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.043655] mdt_io00_041 S ffff8b8cf87c9040 0 118780 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.051016] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.053652] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.058841] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.065723] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.071512] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.077997] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.085569] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.090627] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.096895] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.103515] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.109790] mdt03_105 S ffff8b8cf87ca080 0 118781 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.117144] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.119776] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.124965] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.131850] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.137642] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.144124] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.151699] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.156765] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.163080] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.169700] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.175992] mdt03_106 S ffff8b8cf87cb0c0 0 118782 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.183398] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.186047] [] ? ktime_get_seconds+0x25/0x40 [libcfs] [Fri Apr 12 11:12:51 2019][2937693.192967] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.200754] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.205938] [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.213029] [] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.220102] [] ? class_handle_hash+0xab/0x2f0 [obdclass] [Fri Apr 12 11:12:51 2019][2937693.227247] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.233034] [] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.240385] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.248420] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:51 2019][2937693.256904] [] mdt_object_local_lock+0x50b/0xb20 [mdt] [Fri Apr 12 11:12:51 2019][2937693.264226] [] ? mdt_object_alloc+0x2c0/0x2c0 [mdt] [Fri Apr 12 11:12:51 2019][2937693.270988] [] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.279239] [] ? lustre_msg_bufcount+0x80/0x80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.286359] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.293475] [] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Fri Apr 12 11:12:51 2019][2937693.300639] [] mdt_object_lock_try+0x27/0xb0 [mdt] [Fri Apr 12 11:12:51 2019][2937693.307272] [] mdt_getattr_name_lock+0x1287/0x1c30 [mdt] [Fri Apr 12 11:12:51 2019][2937693.314476] [] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.321150] [] ? __req_capsule_get+0x15f/0x740 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.328246] [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.335406] [] mdt_intent_getattr+0x2b5/0x480 [mdt] [Fri Apr 12 11:12:51 2019][2937693.342130] [] mdt_intent_policy+0x2e8/0xd00 [mdt] [Fri Apr 12 11:12:51 2019][2937693.348756] [] ? mdt_intent_layout+0xcc0/0xcc0 [mdt] [Fri Apr 12 11:12:51 2019][2937693.355589] [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.362484] [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] [Fri Apr 12 11:12:51 2019][2937693.369833] [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [Fri Apr 12 11:12:51 2019][2937693.376416] [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.383704] [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.391426] [] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.397778] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.404919] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.412680] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:12:51 2019][2937693.419971] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.427871] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.434850] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:51 2019][2937693.441300] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:12:51 2019][2937693.447394] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.455904] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.463517] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.468597] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.474868] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.481486] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.487762] mdt_io01_029 S ffff8b8cf6dd8000 0 118783 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.495126] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.497759] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.502989] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.509920] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.515751] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.522260] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.529846] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.534905] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.541176] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.547802] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.554080] mdt_io00_042 S ffff8b8cf6dd9040 0 118784 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.561444] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.564076] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.569268] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.576150] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.581943] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.588429] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.596005] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.601102] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.607376] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.614007] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.620285] mdt_io02_054 S ffff8b8cf6dda080 0 118785 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.627660] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.630291] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.635481] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.642388] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.648221] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.654723] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.662318] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.667377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.673660] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.680315] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.686918] mdt_io03_044 S ffff8b8cf87cc100 0 118786 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.694446] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.697096] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.702298] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.709177] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.714987] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.724565] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.732362] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.737431] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.743720] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.750399] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.756682] mdt03_107 S ffff8b8cf87cd140 0 118787 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.764065] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.766698] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.771892] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.778771] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.784563] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.791046] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.798619] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.803678] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.809953] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:51 2019][2937693.816572] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:51 2019][2937693.822846] mdt_io00_043 S ffff8b8cf87ce180 0 118788 2 0x00000080 [Fri Apr 12 11:12:51 2019][2937693.830202] Call Trace: [Fri Apr 12 11:12:51 2019][2937693.832833] [] schedule+0x29/0x70 [Fri Apr 12 11:12:51 2019][2937693.838020] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.844899] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:51 2019][2937693.850692] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.857170] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:51 2019][2937693.864744] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:51 2019][2937693.869806] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937693.876082] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937693.882713] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937693.888986] mdt_io02_055 S ffff8b8b0a718000 0 118789 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937693.896357] Call Trace: [Fri Apr 12 11:12:52 2019][2937693.898990] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937693.904183] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937693.911078] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937693.916890] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937693.923383] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937693.930973] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937693.936033] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937693.942307] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937693.948940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937693.955219] mdt_io02_056 S ffff8b8b0a719040 0 118790 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937693.962590] Call Trace: [Fri Apr 12 11:12:52 2019][2937693.965223] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937693.970426] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937693.977306] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937693.983095] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937693.989576] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937693.997149] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.002211] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.008489] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.015107] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.021383] mdt_io02_057 S ffff8b8b0a71a080 0 118791 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.028772] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.031411] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.036610] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.043497] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.049307] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.055798] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.063376] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.068438] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.074731] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.081360] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.087640] mdt_io00_044 S ffff8b8af6254100 0 118792 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.095030] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.097669] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.102867] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.109754] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.115550] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.122061] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.129639] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.134700] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.140988] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.147608] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.153880] mdt_io03_045 S ffff8b8af6255140 0 118793 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.161243] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.163874] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.169064] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.175949] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.181743] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.188266] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.195839] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.200897] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.207214] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.213847] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.220130] mdt03_108 S ffff8b8af6256180 0 118794 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.228290] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.230988] [] ? tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.238219] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.243579] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.250650] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.256437] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.263101] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.270695] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.275928] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.282202] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.288996] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.295269] mdt_io00_045 S ffff8b8943bd0000 0 118795 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.302625] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.305255] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.310447] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.317330] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.323121] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.329606] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.337175] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.342748] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.349347] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.356066] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.362345] mdt03_109 S ffff8b8943bd1040 0 118796 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.369706] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.372338] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.377531] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.384414] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.390251] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.396739] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.404311] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.409371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.415647] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.422268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.428549] mdt_io02_058 S ffff8b8943bd2080 0 118797 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.435912] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.438546] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.443734] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.450617] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.456406] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.462890] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.470472] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.475535] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.481812] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.488441] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.494727] mdt_io03_046 S ffff8b8943bd30c0 0 118798 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.502094] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.504741] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.509946] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.516851] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.522660] [] ptlrpc_main+0x1689/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.529229] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.536800] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.541860] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.548135] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.554775] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.561066] mdt03_110 S ffff8b8943bd4100 0 118799 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.568456] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.571098] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.579963] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.587292] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.593109] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.602175] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.609824] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.614882] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.621158] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.627790] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.634078] mdt_io00_046 S ffff8b8943bd5140 0 118800 2 0x00000080 [Fri Apr 12 11:12:52 2019][2937694.641450] Call Trace: [Fri Apr 12 11:12:52 2019][2937694.644092] [] schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.649237] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:12:52 2019][2937694.654651] [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] [Fri Apr 12 11:12:52 2019][2937694.663413] [] ? kiblnd_queue_tx+0x36/0x50 [ko2iblnd] [Fri Apr 12 11:12:52 2019][2937694.670377] [] ? kiblnd_launch_tx+0xd2/0xc20 [ko2iblnd] [Fri Apr 12 11:12:52 2019][2937694.677434] [] ? kiblnd_send+0x222/0xa20 [ko2iblnd] [Fri Apr 12 11:12:52 2019][2937694.684176] [] ? target_bulk_io+0x4a6/0xab0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.691001] [] ? lnet_ni_send+0x44/0xd0 [lnet] [Fri Apr 12 11:12:52 2019][2937694.697297] [] ? lnet_send+0x82/0x1c0 [lnet] [Fri Apr 12 11:12:52 2019][2937694.703403] [] ? LNetPut+0x2cc/0xb50 [lnet] [Fri Apr 12 11:12:52 2019][2937694.709456] [] ? ptl_send_buf+0x146/0x530 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.716078] [] ? ___slab_alloc+0x209/0x4f0 [Fri Apr 12 11:12:52 2019][2937694.722041] [] ? lustre_msg_buf_v2+0x1b0/0x1b0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.729093] [] ? update_curr+0x14c/0x1e0 [Fri Apr 12 11:12:52 2019][2937694.734846] [] ? account_entity_dequeue+0xae/0xd0 [Fri Apr 12 11:12:52 2019][2937694.741380] [] ? dequeue_entity+0x11c/0x5e0 [Fri Apr 12 11:12:52 2019][2937694.747413] [] ? set_next_entity+0x3c/0xe0 [Fri Apr 12 11:12:52 2019][2937694.753351] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:12:52 2019][2937694.759330] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.766214] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:12:52 2019][2937694.775992] [] ? tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.783533] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.791279] [] ? schedule+0x29/0x70 [Fri Apr 12 11:12:52 2019][2937694.796639] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.804733] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.811824] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:12:52 2019][2937694.818275] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:52 2019][2937694.824074] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.830762] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:52 2019][2937694.838349] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:12:52 2019][2937694.843599] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.849876] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:52 2019][2937694.856677] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:52 2019][2937694.862953] mdt_io02_059 S ffff8b8943bd6180 0 118801 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937694.870331] Call Trace: [Fri Apr 12 11:12:53 2019][2937694.872966] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937694.878154] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937694.885048] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937694.892506] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937694.899012] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937694.906595] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937694.911652] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937694.917927] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937694.924545] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937694.930820] mdt_io03_047 S ffff8b86d4300000 0 118802 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937694.938185] Call Trace: [Fri Apr 12 11:12:53 2019][2937694.940816] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937694.946010] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937694.952889] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937694.958681] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937694.965161] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937694.972735] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937694.977795] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937694.984070] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937694.990692] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937694.996967] mdt03_111 S ffff8b86d4301040 0 118803 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.004337] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.006970] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.012158] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.019037] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.024828] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.031323] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.038901] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.043960] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.050238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.056855] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.063129] mdt_io00_047 S ffff8b56b2bab0c0 0 118804 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.070493] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.073124] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.078318] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.085197] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.090992] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.097480] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.105052] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.110111] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.116388] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.123026] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.129299] mdt_io03_048 S ffff8b56b2bac100 0 118805 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.136655] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.139287] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.144474] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.151353] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.157143] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.163627] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.171195] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.176256] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.182534] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.189153] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.195427] mdt_io02_060 S ffff8b8cf729c100 0 118806 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.202782] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.205417] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.210604] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.217487] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.223279] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.229762] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.237332] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.242395] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.248670] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.255291] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.261565] mdt_io02_061 S ffff8b8cf729d140 0 118807 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.268929] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.271564] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.276750] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.283636] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.289446] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.295953] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.303531] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.308594] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.314867] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.321489] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.327761] mdt_io03_049 S ffff8b8cf729b0c0 0 118808 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.335119] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.337750] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.342937] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.349813] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.355605] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.362089] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.369660] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.374719] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.381017] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.387639] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.393917] mdt_io02_062 S ffff8b8cf729e180 0 118809 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.401280] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.403912] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.409118] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.416004] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.421792] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.428272] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.435859] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.440917] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.447196] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.453817] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.460090] mdt_io03_050 S ffff8b8cff016180 0 118810 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.467460] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.470092] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.475281] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.482183] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.487975] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.494463] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.502037] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.507098] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.513373] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.519995] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.526268] mdt_io02_063 S ffff8b8cff012080 0 118811 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.533633] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.536264] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.541462] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.548348] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.554140] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.560616] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.568192] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.573254] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.579528] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.586148] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.592424] mdt_io00_048 S ffff8b8cff0130c0 0 118812 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.599787] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.602417] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.607603] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.614480] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.620272] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.626754] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.634328] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.639389] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.645665] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.652286] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.658557] mdt_io03_051 S ffff8b8cff014100 0 118813 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.665915] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.668561] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.673751] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.680628] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.686419] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.692903] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.700476] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.705553] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.711828] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.718450] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.724737] mdt_io03_052 S ffff8b8cff011040 0 118814 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.732095] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.734729] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.739915] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.746811] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.752599] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.759081] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.766656] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.771717] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.778002] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.784619] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.790892] mdt_io02_064 S ffff8b7a7c299040 0 118815 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.798259] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.800892] [] schedule+0x29/0x70 [Fri Apr 12 11:12:53 2019][2937695.806080] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.812969] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:53 2019][2937695.818780] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.825261] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:53 2019][2937695.832839] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:53 2019][2937695.837897] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.844175] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:53 2019][2937695.850810] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:53 2019][2937695.857087] mdt_io00_049 S ffff8b8cff010000 0 118816 2 0x00000080 [Fri Apr 12 11:12:53 2019][2937695.864456] Call Trace: [Fri Apr 12 11:12:53 2019][2937695.867087] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937695.872294] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937695.879180] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937695.884974] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937695.891459] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937695.899049] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937695.904111] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937695.910424] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937695.917042] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937695.923316] mdt_io02_065 S ffff8b758fbda080 0 118817 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937695.930706] Call Trace: [Fri Apr 12 11:12:54 2019][2937695.933338] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937695.938528] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937695.945422] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937695.951222] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937695.957709] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937695.965285] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937695.970345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937695.976619] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937695.983238] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937695.989511] mdt_io01_030 S ffff8b758fbdc100 0 118818 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937695.996869] Call Trace: [Fri Apr 12 11:12:54 2019][2937695.999497] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.004691] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.011572] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.017367] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.023868] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.031445] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.036507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.042781] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.049403] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.055674] mdt_io01_031 S ffff8b758fbdd140 0 118819 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.063031] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.065663] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.070851] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.077736] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.083527] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.090014] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.097590] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.102649] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.108918] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.115537] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.121812] mdt_io02_066 S ffff8b758fbde180 0 118820 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.129176] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.131807] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.136999] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.143880] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.149676] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.156166] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.163734] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.168795] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.175070] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.181693] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.187966] mdt_io03_053 S ffff8b758fbdb0c0 0 118821 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.195322] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.197952] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.203141] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.210018] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.215812] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.222302] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.229871] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.234932] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.241208] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.247830] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.254102] mdt_io01_032 S ffff8b758fbd9040 0 118822 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.261459] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.264091] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.269283] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.276164] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.281954] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.288435] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.296008] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.301069] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.307346] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.313966] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.320258] mdt_io03_054 S ffff8b7429384100 0 118823 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.327630] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.330266] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.335453] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.342338] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.348131] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.354628] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.362209] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.367270] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.373554] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.380222] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.386498] mdt_io01_033 S ffff8b7429380000 0 118824 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.393863] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.396497] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.401690] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.408567] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.414357] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.420839] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.428414] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.433475] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.439748] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.446373] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.452644] mdt_io01_034 S ffff8b7429386180 0 118825 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.460007] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.462639] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.467828] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.474712] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.480505] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.486989] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.494566] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.499630] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.505907] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.512536] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.518816] mdt_io02_067 S ffff8b7429381040 0 118826 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.526179] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.528813] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.534019] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.540903] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.546693] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.553174] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.560748] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.565819] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.572092] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.578712] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.584988] mdt_io01_035 S ffff8b74293830c0 0 118827 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.592375] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.595010] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.600214] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.607129] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.612959] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.619456] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.627617] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.632842] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.639133] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.645752] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.652023] mdt_io00_050 S ffff8b7cf3304100 0 118828 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.659410] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.662066] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.667284] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.674174] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.679979] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.686469] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.694055] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.699127] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.705416] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.712037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.718317] mdt_io03_055 S ffff8b7cf33030c0 0 118829 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.725699] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.728334] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.733525] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.740417] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.746233] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.752742] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.760352] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.765415] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.771690] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.778309] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.784585] mdt_io02_068 S ffff8b7cf3305140 0 118830 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.791977] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.794618] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.799811] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.806687] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:54 2019][2937696.812492] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.819020] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.826983] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:54 2019][2937696.832256] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.838551] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:54 2019][2937696.845266] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:54 2019][2937696.851546] mdt_io01_036 S ffff8b7cf3300000 0 118831 2 0x00000080 [Fri Apr 12 11:12:54 2019][2937696.858938] Call Trace: [Fri Apr 12 11:12:54 2019][2937696.861579] [] schedule+0x29/0x70 [Fri Apr 12 11:12:54 2019][2937696.866780] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:54 2019][2937696.874248] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937696.880219] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937696.886702] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937696.894285] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937696.899344] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937696.905620] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937696.912240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937696.918513] mdt_io00_051 S ffff8b7cf640a080 0 118832 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937696.925876] Call Trace: [Fri Apr 12 11:12:55 2019][2937696.928510] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937696.933702] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937696.940611] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937696.946462] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937696.952959] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937696.960544] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937696.965604] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937696.971883] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937696.978510] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937696.984782] mdt_io03_056 S ffff8b7cf6408000 0 118833 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937696.992154] Call Trace: [Fri Apr 12 11:12:55 2019][2937696.994788] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937696.999981] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.006869] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.012657] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.019145] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.026724] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.031782] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.038058] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.044679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.050953] mdt_io00_052 S ffff8b7cf640e180 0 118834 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.058328] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.060970] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.066168] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.073050] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.078845] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.085337] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.092911] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.097972] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.104250] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.110898] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.117168] mdt_io01_037 S ffff8b657562a080 0 118835 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.124531] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.127164] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.132354] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.139239] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.145037] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.151518] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.159092] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.164154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.170455] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.177076] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.183348] mdt_io00_053 S ffff8b657562e180 0 118836 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.190710] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.193345] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.198536] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.205417] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.211210] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.217705] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.225282] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.230366] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.236645] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.243263] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.249539] mdt_io02_069 S ffff8b657562d140 0 118837 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.256900] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.259532] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.264723] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.271608] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.277402] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.283897] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.291470] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.296530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.302808] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.309425] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.315700] mdt_io03_057 S ffff8b65a1299040 0 118838 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.323063] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.325694] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.330895] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.337777] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.343573] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.350068] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.357644] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.362702] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.368978] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.375599] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.381874] mdt_io03_058 S ffff8b64b83ea080 0 118839 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.389233] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.391866] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.397080] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.403977] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.409780] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.416265] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.423840] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.428897] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.435174] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.441794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.448074] mdt_io01_038 S ffff8b64b83e9040 0 118840 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.455440] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.458072] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.463265] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.470167] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.475952] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.482436] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.490012] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.495073] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.501347] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.507968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.514240] mdt_io03_059 S ffff8b64b83ec100 0 118841 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.521605] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.524247] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.529446] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.536328] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.542121] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.548608] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.556186] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.561261] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.567540] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.574163] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.580435] mdt_io01_039 S ffff8b64b83eb0c0 0 118842 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.587792] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.590422] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.595617] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.602495] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.608289] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.614772] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.622345] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.627405] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.633697] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.640321] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.646590] mdt_io03_060 S ffff8b64b83ed140 0 118843 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.653944] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.656599] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.661787] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.668670] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.674456] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.680940] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.688641] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.693700] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.699976] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.706600] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.712876] mdt_io03_061 S ffff8b64b83ee180 0 118844 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.720247] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.722882] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.728074] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.734955] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.740743] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.747227] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.754801] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.759901] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.766182] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.772802] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.779073] mdt_io01_040 S ffff8b64b83e8000 0 118845 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.786438] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.789071] [] schedule+0x29/0x70 [Fri Apr 12 11:12:55 2019][2937697.794260] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.801144] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:55 2019][2937697.806931] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.813413] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:55 2019][2937697.820993] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:55 2019][2937697.826058] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.832335] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:55 2019][2937697.838953] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:55 2019][2937697.845225] mdt_io03_062 S ffff8b5cfc122080 0 118846 2 0x00000080 [Fri Apr 12 11:12:55 2019][2937697.852581] Call Trace: [Fri Apr 12 11:12:55 2019][2937697.855215] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937697.860407] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937697.867287] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937697.873080] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937697.879559] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937697.887134] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937697.892253] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937697.898533] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937697.905151] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937697.911423] mdt_io01_041 S ffff8b5cfc121040 0 118847 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937697.918781] Call Trace: [Fri Apr 12 11:12:56 2019][2937697.921415] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937697.926607] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937697.933511] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937697.939321] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937697.945814] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937697.953402] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937697.958461] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937697.964764] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937697.971385] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937697.977685] mdt_io00_054 S ffff8b5cfc1230c0 0 118848 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937697.985045] Call Trace: [Fri Apr 12 11:12:56 2019][2937697.987681] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937697.992891] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937697.999768] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.005558] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.012064] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.019643] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.024712] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.030998] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.037620] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.044083] mdt_io02_070 S ffff8b5cfc124100 0 118849 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.051713] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.054358] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.059556] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.066439] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.072240] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.078724] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.086299] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.091359] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.097636] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.104258] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.110530] mdt_io01_042 S ffff8b7014b2b0c0 0 118850 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.117891] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.120526] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.125736] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.132645] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.138446] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.144944] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.152541] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.157608] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.164570] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.171377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.177646] mdt_io03_063 S ffff8b7014b2c100 0 118851 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.185016] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.187649] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.192841] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.199742] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.205542] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.212065] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.219640] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.224702] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.230980] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.237620] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.243895] mdt_io02_071 S ffff8b7014b29040 0 118852 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.251276] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.253918] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.259113] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.266001] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.271795] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.278289] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.285864] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.290928] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.297219] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.303837] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.310463] mdt_io00_055 S ffff8b7014b2e180 0 118853 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.318374] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.321012] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.326220] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.333121] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.338921] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.345403] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.353007] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.358067] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.364342] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.370973] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.378582] mdt_io00_056 S ffff8b7014b2d140 0 118854 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.386201] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.388837] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.394032] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.400916] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.406727] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.413217] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.420789] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.425860] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.432135] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.439011] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.445353] mdt_io02_072 S ffff8b65a129e180 0 118856 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.452713] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.455354] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.460547] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.467446] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.473238] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.479729] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.487306] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.492377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.498653] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.505281] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.511554] mdt_io00_057 S ffff8b65a129c100 0 118857 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.518918] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.521554] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.526752] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.533676] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.539485] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.545968] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.553550] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.558663] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.564935] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.571556] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.577837] mdt_io01_043 S ffff8b7014b2a080 0 118858 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.585210] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.587843] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.593055] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.599942] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.605755] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.612244] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.619843] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.624902] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.631177] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.637797] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.644071] mdt_io02_073 S ffff8b7014b28000 0 118859 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.651434] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.654070] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.659270] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.666180] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.671984] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.678484] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.686068] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.691134] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.697411] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.704260] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.710934] mdt_io03_064 S ffff8b8a87379040 0 118860 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.718300] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.720942] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.726133] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.733014] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.738806] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.745293] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.752871] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.757930] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.764214] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.770833] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.777106] mdt_io00_058 S ffff8b8a8737c100 0 118861 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.784470] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.787101] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.792317] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.799225] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:56 2019][2937698.805039] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.811526] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:56 2019][2937698.819106] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:56 2019][2937698.824170] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.830446] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:56 2019][2937698.837068] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:56 2019][2937698.843340] mdt_io02_074 S ffff8b8a87378000 0 118862 2 0x00000080 [Fri Apr 12 11:12:56 2019][2937698.850703] Call Trace: [Fri Apr 12 11:12:56 2019][2937698.853345] [] schedule+0x29/0x70 [Fri Apr 12 11:12:56 2019][2937698.858545] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937698.865425] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937698.871219] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937698.877711] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937698.885308] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937698.890377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937698.896652] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937698.903271] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937698.909544] mdt_io03_065 S ffff8b8a8737e180 0 118863 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937698.916920] Call Trace: [Fri Apr 12 11:12:57 2019][2937698.919557] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937698.924747] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937698.931633] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937698.937422] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937698.943904] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937698.951492] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937698.956567] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937698.962840] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937698.969458] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937698.975735] mdt_io00_059 S ffff8b8a8737d140 0 118864 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937698.983130] Call Trace: [Fri Apr 12 11:12:57 2019][2937698.985764] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937698.990978] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937698.997893] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.003696] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.010184] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.017764] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.022825] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.029108] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.035732] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.042008] mdt_io03_066 S ffff8b8a8737b0c0 0 118865 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.049373] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.052007] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.057201] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.064086] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.069913] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.076395] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.083968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.089037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.095312] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.101975] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.108253] mdt_io03_067 S ffff8b8a8737a080 0 118867 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.115614] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.118248] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.123453] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.130341] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.136155] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.142637] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.150233] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.155306] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.161581] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.168200] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.174475] mdt_io00_060 S ffff8b6cfa3ce180 0 118868 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.181838] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.184472] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.189672] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.196552] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.202348] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.208845] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.216424] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.221484] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.227825] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.234468] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.240751] mdt_io03_068 S ffff8b6cfa3cd140 0 118869 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.248123] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.250756] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.255946] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.262849] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.268646] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.275137] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.282727] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.287786] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.294071] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.300701] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.306981] mdt_io01_044 S ffff8b6cfa3ca080 0 118870 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.314363] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.317116] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.322309] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.329199] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.334993] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.341504] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.349090] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.354159] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.360434] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.367053] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.373324] mdt_io00_061 S ffff8b6cfa3c9040 0 118871 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.380681] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.383314] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.388529] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.395437] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.401244] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.407737] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.415314] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.420372] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.426658] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.433279] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.439560] mdt_io03_069 S ffff8b6cfb20d140 0 118872 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.446920] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.449554] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.454744] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.461625] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.467417] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.473900] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.481472] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.486533] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.492808] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.499428] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.505704] mdt_io01_045 S ffff8b6cfb209040 0 118873 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.513085] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.515753] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.520944] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.527828] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.533647] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.540163] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.547752] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.552811] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.559086] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.565707] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.571997] mdt_io02_075 S ffff8b6cfb208000 0 118874 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.579379] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.582021] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.587221] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.594100] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.599890] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.606371] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.613946] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.619004] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.625277] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.631897] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.638169] mdt_io03_070 S ffff8b6cfb20a080 0 118875 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.645549] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.648183] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.653428] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.660311] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.666113] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.672600] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.680200] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.685267] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.691541] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.698160] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.704434] mdt_io00_062 S ffff8b6cf74c9040 0 118876 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.711788] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.714421] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.719612] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.726493] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.732305] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.738785] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.746361] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.751428] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.757708] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.764343] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.770623] mdt_io02_076 S ffff8b6cf74cb0c0 0 118877 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.778004] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.780642] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.785847] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.792730] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.798548] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.805063] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.812652] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:57 2019][2937699.817711] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.823989] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:57 2019][2937699.830607] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:57 2019][2937699.836886] mdt_io00_063 S ffff8b6cf74ce180 0 118879 2 0x00000080 [Fri Apr 12 11:12:57 2019][2937699.844246] Call Trace: [Fri Apr 12 11:12:57 2019][2937699.846878] [] schedule+0x29/0x70 [Fri Apr 12 11:12:57 2019][2937699.852099] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:57 2019][2937699.859005] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:57 2019][2937699.864810] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937699.871297] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937699.878885] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937699.883965] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937699.890243] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937699.896876] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937699.903182] mdt_io01_046 S ffff8b6cf74c8000 0 118880 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937699.910557] Call Trace: [Fri Apr 12 11:12:58 2019][2937699.913190] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937699.918380] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937699.925284] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937699.931134] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937699.937620] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937699.945198] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937699.950263] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937699.956538] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937699.963160] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937699.969431] mdt_io03_071 S ffff8b6cf74ca080 0 118881 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937699.976786] Call Trace: [Fri Apr 12 11:12:58 2019][2937699.979420] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937699.984611] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937699.991494] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937699.997289] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.003774] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.011385] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.016444] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.022728] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.029352] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.035633] mdt_io01_047 S ffff8b6cfacca080 0 118882 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.043003] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.045637] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.050828] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.057711] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.063513] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.069999] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.077583] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.082643] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.088919] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.095537] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.101830] mdt_io02_077 S ffff8b6cfacc9040 0 118883 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.109193] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.111833] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.117027] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.123906] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.129703] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.136189] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.143772] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.148829] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.155104] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.161739] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.168044] mdt_io03_072 S ffff8b6cfacce180 0 118884 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.175458] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.178103] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.183311] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.190202] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.196003] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.202490] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.210061] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.215143] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.221438] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.228067] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.234335] mdt_io02_078 S ffff8b6cfaccd140 0 118885 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.241701] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.244339] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.249534] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.256414] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.262207] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.268697] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.276275] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.281335] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.287612] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.294233] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.300505] mdt_io02_079 S ffff8b6cfaccc100 0 118886 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.307861] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.310494] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.315683] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.322568] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.328357] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.334838] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.342415] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.347476] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.353759] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.360390] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.366672] mdt_io01_048 S ffff8b6cfc4b8000 0 118887 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.374042] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.376676] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.381868] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.388749] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.394604] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.401096] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.408671] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.413733] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:12:58 2019][2937700.420009] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.426284] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.432906] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.439179] mdt_io03_073 S ffff8b6cfc4bc100 0 118888 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.446575] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.449218] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.454409] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.461291] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.467087] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.473570] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.481145] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.486203] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.492477] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.499100] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.505373] mdt_io00_064 S ffff8b6cfc4bd140 0 118889 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.512727] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.515361] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.520549] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.527427] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.533237] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.539729] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.547327] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.552386] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.558663] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.565283] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.571556] mdt_io00_065 S ffff8b6cfc4bb0c0 0 118890 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.578956] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.581596] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.586788] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.593681] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.599491] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.605994] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.613566] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.618626] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.624914] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.631552] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.637858] mdt_io02_080 S ffff8b6cfc4be180 0 118891 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.645237] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.647870] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.653078] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.659967] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.665854] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.672341] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.679919] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.684980] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.691257] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.697876] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.704151] mdt_io03_074 S ffff8b6c73f90000 0 118892 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.711520] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.714156] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.719343] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.726228] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.732020] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.738507] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.746082] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.751143] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.757419] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.764037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.770314] mdt_io00_066 S ffff8b6c73f91040 0 118893 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.777677] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.780309] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.785502] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.792382] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.798177] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.804668] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.812248] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:58 2019][2937700.817305] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.823582] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:58 2019][2937700.830201] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:58 2019][2937700.836475] mdt_io02_081 S ffff8b6c73f92080 0 118894 2 0x00000080 [Fri Apr 12 11:12:58 2019][2937700.843849] Call Trace: [Fri Apr 12 11:12:58 2019][2937700.846480] [] schedule+0x29/0x70 [Fri Apr 12 11:12:58 2019][2937700.851674] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.858552] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:58 2019][2937700.864350] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:58 2019][2937700.870840] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937700.878418] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937700.883479] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937700.889753] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937700.896371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937700.902648] mdt_io03_075 S ffff8b6c73f930c0 0 118895 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937700.910034] Call Trace: [Fri Apr 12 11:12:59 2019][2937700.912670] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937700.917872] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937700.924759] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937700.930550] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937700.937042] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937700.944616] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937700.949673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937700.955952] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937700.962570] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937700.968844] mdt_io02_082 S ffff8b6c73f94100 0 118896 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937700.976206] Call Trace: [Fri Apr 12 11:12:59 2019][2937700.978840] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937700.984032] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937700.990914] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937700.996714] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.003202] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.010780] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.015839] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.022116] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.028759] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.035034] mdt_io03_076 S ffff8b6c73f95140 0 118897 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.042403] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.045038] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.050237] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.057119] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.062916] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.069406] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.076984] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.082043] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.088321] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.094941] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.101215] mdt_io01_049 S ffff8b69d5a3a080 0 118898 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.108575] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.111209] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.116401] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.123281] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.129078] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.135559] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.143136] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.148199] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.154499] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.161122] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.167395] mdt_io00_067 S ffff8b69d5a3d140 0 118899 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.174755] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.177389] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.182585] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.189470] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.195269] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.201761] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.209336] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.214396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.220672] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.227291] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.233567] mdt_io02_083 S ffff8b69d5a3c100 0 118900 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.240927] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.243560] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.248759] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.255642] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.261434] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.267948] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.275526] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.280585] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.286861] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.293480] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.299753] mdt_io03_077 S ffff8b69cebdd140 0 118901 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.307126] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.309758] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.314957] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.321843] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.327636] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.334131] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.341704] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.346763] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.353039] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.359659] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.365936] mdt_out02_002 S ffff8b657562c100 0 118902 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.373297] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.375929] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.381148] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.388044] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.393845] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.400335] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.407910] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.412974] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.419246] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.425867] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.432141] mdt_io02_084 S ffff8b657562b0c0 0 118903 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.439503] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.442135] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.447332] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.454217] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.460014] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.466506] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.474086] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.479152] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.485427] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.492049] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.498323] mdt_io03_078 S ffff8b65a129a080 0 118904 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.505683] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.508315] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.513513] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.520399] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.526199] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.532694] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.540271] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.545331] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.551607] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.558227] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.564502] mdt_io02_085 S ffff8b65a129b0c0 0 118905 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.571888] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.574523] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.579724] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.586604] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.592403] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.598892] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.606468] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.611529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.617804] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.624424] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.630701] mdt_io00_068 S ffff8b65a129d140 0 118906 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.638060] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.640695] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.645889] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.652776] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.658577] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.665066] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.672640] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.677698] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.683978] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.690619] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.696897] mdt_io03_079 S ffff8b5e5e48d140 0 118907 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.704258] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.706893] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.712093] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.718997] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.724794] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.731284] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.738852] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.743916] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.750216] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.756840] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.763112] mdt_io02_086 S ffff8b5e5e48c100 0 118908 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.770481] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.773116] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.778304] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.785188] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.790978] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.797467] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.805044] [] kthread+0xd1/0xe0 [Fri Apr 12 11:12:59 2019][2937701.810108] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.816382] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:12:59 2019][2937701.823000] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:12:59 2019][2937701.829273] mdt_io00_069 S ffff8b5e5e488000 0 118909 2 0x00000080 [Fri Apr 12 11:12:59 2019][2937701.836637] Call Trace: [Fri Apr 12 11:12:59 2019][2937701.839268] [] schedule+0x29/0x70 [Fri Apr 12 11:12:59 2019][2937701.844470] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.851355] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:12:59 2019][2937701.857155] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.863647] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:12:59 2019][2937701.871223] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937701.876283] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937701.882561] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937701.889180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937701.895457] mdt_io03_080 S ffff8b5e5e48a080 0 118910 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937701.902816] Call Trace: [Fri Apr 12 11:13:00 2019][2937701.905448] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937701.910640] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937701.917521] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937701.923311] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937701.929800] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937701.937380] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937701.942437] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937701.948715] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937701.955334] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937701.961610] mdt_io02_087 S ffff8b5e5e48b0c0 0 118911 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937701.968971] Call Trace: [Fri Apr 12 11:13:00 2019][2937701.971601] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937701.976796] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937701.983677] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937701.989480] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937701.995972] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.003550] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.008609] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.014886] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.021505] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.027780] mdt_io02_088 S ffff8b5e5e489040 0 118912 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.035142] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.037773] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.042971] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.049858] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.055652] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.062146] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.069722] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.074780] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.081058] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.087677] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.093955] mdt_io03_081 S ffff8b88d036e180 0 118913 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.101313] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.103946] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.109154] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.116038] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.121839] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.128334] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.135910] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.140969] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.147247] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.153866] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.160142] mdt_io01_050 S ffff8b88d0368000 0 118914 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.167511] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.170143] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.175338] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.182217] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.188013] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.194506] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.202085] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.207152] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.213428] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.220047] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.226321] mdt_io02_089 S ffff8b69d5a39040 0 118915 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.233691] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.236323] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.241519] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.248396] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.254189] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.260676] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.268253] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.273313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.279588] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.286209] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.292486] mdt_io00_070 S ffff8b69d5a38000 0 118916 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.299853] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.302487] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.307685] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.314570] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.320360] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.326847] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.334424] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.339483] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.345774] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.352426] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.358698] mdt_io03_082 S ffff8b69d5a3e180 0 118917 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.366061] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.368693] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.373886] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.380767] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.386557] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.393044] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.400621] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.405680] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.411958] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.418577] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.424855] mdt_io02_090 S ffff8b69d5a3b0c0 0 118918 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.432214] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.434848] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.440038] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.446918] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.452712] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.459198] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.466777] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.471863] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.478142] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.484768] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.491041] mdt_io02_091 S ffff8b5c6b39a080 0 118923 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.498404] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.501037] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.506229] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.513113] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.518906] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.525396] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.532974] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.538033] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.544311] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.550929] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.557205] mdt_io02_092 S ffff8b7cf640d140 0 118924 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.564565] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.567200] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.572395] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.579283] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.585077] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.591570] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.599146] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.604205] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.610484] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.617121] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.623393] mdt_io03_083 S ffff8b78aca22080 0 118925 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.630755] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.633387] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.638585] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.645472] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.651274] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.657767] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.665487] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.670561] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.676837] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.683454] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.689731] mdt_io00_071 S ffff8b78aca21040 0 118928 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.697102] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.699732] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.704928] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.711814] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.717606] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.724096] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.731671] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.736731] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.743006] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.749626] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.755900] mdt_io01_051 S ffff8b78aca230c0 0 118929 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.763262] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.765895] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.771098] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.777979] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.783774] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.790265] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.797842] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:00 2019][2937702.802898] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.809179] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:00 2019][2937702.815795] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:00 2019][2937702.822074] mdt_io01_052 S ffff8b78aca20000 0 118930 2 0x00000080 [Fri Apr 12 11:13:00 2019][2937702.829433] Call Trace: [Fri Apr 12 11:13:00 2019][2937702.832066] [] schedule+0x29/0x70 [Fri Apr 12 11:13:00 2019][2937702.837259] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.844141] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:00 2019][2937702.849932] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:00 2019][2937702.856422] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937702.863998] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937702.869057] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937702.875335] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937702.881954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937702.888228] mdt_io02_093 S ffff8b7a7c29e180 0 118931 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937702.895595] Call Trace: [Fri Apr 12 11:13:01 2019][2937702.898231] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937702.903422] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937702.910305] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937702.916094] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937702.922583] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937702.930160] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937702.935226] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937702.941506] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937702.948123] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937702.954417] mdt_io03_084 S ffff8b6fc1f05140 0 118932 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937702.961785] Call Trace: [Fri Apr 12 11:13:01 2019][2937702.964420] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937702.969614] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937702.976532] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937702.982333] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937702.988816] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937702.996391] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.001450] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.007727] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.014347] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.020619] mdt_io00_072 S ffff8b73cf700000 0 118933 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.027993] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.030627] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.035819] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.042699] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.048490] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.054982] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.062577] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.067640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.073917] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.080536] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.086810] mdt_io02_094 S ffff8b73cf706180 0 118934 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.094175] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.096816] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.102010] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.108894] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.114703] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.121186] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.128768] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.133828] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.140106] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.146725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.153001] mdt_io00_073 S ffff8b73cf705140 0 118935 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.160360] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.162995] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.168188] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.175070] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.180873] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.187367] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.194940] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.199999] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.206321] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.212946] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.219394] mdt_io00_074 S ffff8b73cf704100 0 118937 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.226785] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.229429] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.234636] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.241538] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.247340] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.253824] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.261396] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.266461] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.272755] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.279384] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.285652] mdt_io03_085 S ffff8b73cf7030c0 0 118938 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.293015] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.295648] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.300853] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.307749] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.313546] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.320029] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.327604] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.332666] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.338941] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.345559] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.351834] mdt_io02_095 S ffff8b6fc1f01040 0 118939 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.359205] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.361836] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.367030] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.373910] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.379701] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.386193] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.393774] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.398836] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.405115] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.411738] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.418011] mdt_io01_053 S ffff8b6fc1f030c0 0 118940 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.425366] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.427999] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.433195] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.440073] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.445881] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.452361] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.459937] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.465016] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.471291] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.477913] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.484204] mdt_io02_096 S ffff8b6fc1f00000 0 118944 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.491565] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.494198] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.499390] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.506272] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.512065] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.518573] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.526156] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.531223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.537497] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.544144] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.550416] mdt_io00_075 S ffff8b6fc1f02080 0 118945 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.557781] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.560412] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.565643] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.572566] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.578357] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.584853] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.592437] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.597497] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.603774] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.610393] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.616667] mdt_io00_076 S ffff8b6fc1f04100 0 118947 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.624038] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.626668] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.631863] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.638745] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.644536] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.651016] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.658599] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.663677] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.669954] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.676573] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.682850] mdt_io02_097 S ffff8b6fc1f06180 0 118949 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.690210] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.692843] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.698035] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.704927] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.710716] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.717195] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.724774] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.729835] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.736116] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.742744] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.749019] mdt_io01_054 S ffff8b5b7ffea080 0 118950 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.756382] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.759013] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.764202] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.771088] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.776877] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.783380] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.790949] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:01 2019][2937703.796009] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.802285] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:01 2019][2937703.808906] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:01 2019][2937703.815179] mdt_io02_098 S ffff8b5b7ffe9040 0 118951 2 0x00000080 [Fri Apr 12 11:13:01 2019][2937703.822536] Call Trace: [Fri Apr 12 11:13:01 2019][2937703.825165] [] schedule+0x29/0x70 [Fri Apr 12 11:13:01 2019][2937703.830356] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.837240] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:01 2019][2937703.843030] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.849513] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:01 2019][2937703.857089] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937703.862148] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937703.868423] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937703.875044] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937703.881321] mdt_io02_099 S ffff8b5b7ffe8000 0 118952 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937703.888701] Call Trace: [Fri Apr 12 11:13:02 2019][2937703.891339] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937703.896533] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937703.903423] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937703.909232] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937703.915710] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937703.923286] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937703.928347] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937703.934631] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937703.941731] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937703.948081] mdt_io00_077 S ffff8b5b7ffed140 0 118953 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937703.955442] Call Trace: [Fri Apr 12 11:13:02 2019][2937703.958076] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937703.963271] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937703.970162] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937703.975990] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937703.982469] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937703.990045] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937703.995107] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.001381] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.008002] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.014276] mdt_io00_078 S ffff8b65296a1040 0 118955 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.021640] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.024269] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.029462] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.036345] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.042139] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.048627] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.056200] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.061258] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.067534] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.074157] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.080428] mdt_io02_100 S ffff8b6da99e30c0 0 118956 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.087785] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.090435] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.095620] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.102496] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.108290] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.114767] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.122343] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.127402] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.133672] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.140291] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.146582] mdt_io02_101 S ffff8b6cf9b39040 0 118957 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.153941] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.156570] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.161757] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.168633] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.174423] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.180908] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.188480] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.193541] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.199820] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.206438] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.212711] mdt_io00_079 S ffff8b7ba5af8000 0 118958 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.220068] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.222697] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.227888] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.234772] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.240564] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.247050] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.254628] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.259699] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.265970] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.272594] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.278868] mdt_io01_055 S ffff8b7ba5af9040 0 118960 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.286249] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.288882] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.294083] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.300964] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.306757] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.313241] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.320844] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.325922] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.332198] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.338817] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.345101] mdt_io00_080 S ffff8b73cf701040 0 118961 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.352480] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.355116] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.360314] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.367192] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.372985] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.379465] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.387091] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.392154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.398428] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.405047] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.411322] mdt_io00_081 S ffff8b73cf702080 0 118962 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.418684] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.421318] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.426505] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.433381] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.439176] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.445662] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.453237] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.458301] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.464588] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.471223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.477493] mdt_io01_056 S ffff8b6c16294100 0 118963 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.484857] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.487489] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.492679] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.499570] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.505362] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.511845] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.519418] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.524477] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.530753] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.537373] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.543648] mdt_io01_057 S ffff8b8cbaaa1040 0 118964 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.551011] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.553641] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.558831] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.565707] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.571499] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.577980] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.585555] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.590620] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.596891] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.603512] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.609785] mdt_io01_058 D ffff8b8cbaaa5140 0 118965 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.617149] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.619782] [] schedule_preempt_disabled+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.626457] [] ? tgt_lookup_reply_by_xid+0x3c/0x60 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.633916] [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.640840] [] ? tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.648110] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.655867] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:13:02 2019][2937704.663135] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.671188] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.678171] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:02 2019][2937704.684617] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:02 2019][2937704.690672] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.697354] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.704930] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.710165] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.716440] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.723232] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.729499] mdt_io00_082 S ffff8b56b2bad140 0 118966 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.736864] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.739497] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.744700] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.751774] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.757601] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.764102] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.771984] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.777531] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.783933] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.790550] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.796823] mdt_io01_059 S ffff8b56b2ba8000 0 118967 2 0x00000080 [Fri Apr 12 11:13:02 2019][2937704.804189] Call Trace: [Fri Apr 12 11:13:02 2019][2937704.806819] [] schedule+0x29/0x70 [Fri Apr 12 11:13:02 2019][2937704.812013] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.818892] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:02 2019][2937704.824712] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.831220] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:02 2019][2937704.838793] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:02 2019][2937704.843853] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.850130] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:02 2019][2937704.856751] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:02 2019][2937704.863026] mdt_io02_102 S ffff8b56b2baa080 0 118968 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937704.870423] Call Trace: [Fri Apr 12 11:13:03 2019][2937704.873082] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937704.878273] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937704.885154] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937704.890943] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937704.897427] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937704.905011] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937704.910073] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937704.916339] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937704.922963] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937704.929238] mdt_io01_060 S ffff8b56b2ba9040 0 118969 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937704.936601] Call Trace: [Fri Apr 12 11:13:03 2019][2937704.939230] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937704.944421] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937704.951303] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937704.957095] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937704.963577] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937704.971159] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937704.976232] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937704.982516] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937704.989137] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937704.995420] mdt_io00_083 S ffff8b5cfbe6b0c0 0 118971 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.002790] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.005422] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.010626] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.017506] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.023317] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.029804] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.037375] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.042435] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.048732] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.055352] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.061656] mdt_io01_061 S ffff8b5cfbe6d140 0 118972 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.069030] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.071667] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.076883] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.083789] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.089584] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.096089] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.103671] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.108727] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.115005] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.121624] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.127899] mdt_io01_062 S ffff8b5cfbe68000 0 118973 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.135262] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.137892] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.143084] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.149966] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.155757] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.162238] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.169812] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.174875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.181171] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.187786] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.194061] mdt_io00_084 S ffff8b5cfbe6a080 0 118974 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.201424] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.204059] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.209276] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.216166] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.222989] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.229824] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.237401] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.242482] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.248792] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.255417] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.261690] mdt_io02_103 S ffff8b5cfbe6e180 0 118975 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.269063] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.271705] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.276896] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.283776] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.289569] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.296058] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.303660] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.308723] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.315021] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.321640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.327913] mdt_io02_104 S ffff8b5cfbe69040 0 118976 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.335276] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.337907] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.343102] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.349991] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.355807] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.362292] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.369925] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.374984] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.381271] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.387887] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.394163] mdt_io02_105 S ffff8b5cfbe6c100 0 118977 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.401526] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.404161] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.409350] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.416234] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.422026] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.428515] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.436085] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.441143] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.447412] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.454033] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.460307] mdt_io02_106 S ffff8b8b64a15140 0 118978 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.467670] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.470302] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.475493] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.482375] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.488170] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.494658] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.502232] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.507291] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.513567] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.520188] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.526461] mdt_io02_107 S ffff8b79c36bc100 0 118979 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.533825] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.536454] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.541653] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.548544] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.554361] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.560860] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.568449] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.573507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.579783] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.586426] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.592696] mdt_io00_085 S ffff8b79c36bb0c0 0 118980 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.600058] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.602699] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.607893] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.614773] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.620566] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.627055] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.634647] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.639703] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.645982] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.652606] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.658892] mdt_io00_086 S ffff8b79c36be180 0 118981 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.666254] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.668890] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.674113] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.680997] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.686816] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.693313] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.700891] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.705964] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.712241] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.718862] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.725143] mdt_io01_063 S ffff8b69cebdb0c0 0 118982 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.732512] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.735144] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.740338] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.747221] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.753019] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.759534] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.767146] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.772204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.778478] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.785119] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.791389] mdt_io00_087 S ffff8b69cebda080 0 118983 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.798745] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.801378] [] schedule+0x29/0x70 [Fri Apr 12 11:13:03 2019][2937705.806568] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.813449] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:03 2019][2937705.819242] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.825724] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:03 2019][2937705.833307] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:03 2019][2937705.838369] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.844643] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:03 2019][2937705.851266] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:03 2019][2937705.857535] mdt_io00_088 S ffff8b75446fa080 0 118991 2 0x00000080 [Fri Apr 12 11:13:03 2019][2937705.864893] Call Trace: [Fri Apr 12 11:13:03 2019][2937705.867534] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937705.872726] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937705.879606] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937705.885400] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937705.891897] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937705.899480] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937705.904569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937705.910925] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937705.917609] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937705.923898] mdt_io02_108 S ffff8b75446fb0c0 0 118996 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937705.931262] Call Trace: [Fri Apr 12 11:13:04 2019][2937705.933896] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937705.939095] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937705.945979] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937705.951769] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937705.958251] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937705.965821] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937705.970885] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937705.977163] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937705.983788] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937705.990062] mdt_io01_064 S ffff8b75446fc100 0 118997 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937705.997437] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.000081] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.005275] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.012157] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.017954] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.024438] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.032013] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.037075] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.043351] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.050047] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.056321] mdt_io02_109 S ffff8b75446fd140 0 118999 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.063692] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.066325] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.071518] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.078397] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.084194] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.090676] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.098253] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.103311] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.109588] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.116208] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.122482] mdt_io02_110 S ffff8b78aca24100 0 119000 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.129845] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.132475] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.137666] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.144571] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.150360] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.156840] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.164412] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.169475] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.175748] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.182410] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.188706] mdt_io01_065 S ffff8b69cebde180 0 119003 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.196165] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.198794] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.203985] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.210869] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.216660] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.223142] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.230714] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.235775] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.242050] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.248673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.254945] mdt_io00_089 S ffff8b69cebd8000 0 119004 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.262301] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.264931] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.270122] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.277004] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.282796] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.289282] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.296859] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.301922] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.308206] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.314828] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.321100] mdt_io01_066 S ffff8b69cebd9040 0 119005 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.328526] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.331158] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.336347] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.343229] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.349022] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.355503] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.363077] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.368136] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.374412] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.381033] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.387306] mdt_io01_067 S ffff8b69cebdc100 0 119006 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.394679] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.397312] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.402500] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.409384] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.415176] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.421662] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.429239] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.434299] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.440576] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.447216] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.453487] mdt_io00_090 S ffff8b5ddf621040 0 119007 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.460851] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.463492] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.468734] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.475618] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.481419] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.487898] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.495472] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.500532] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.506807] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.513429] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.519703] mdt_io00_091 S ffff8b56b2bae180 0 119008 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.527066] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.529697] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.534886] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.541769] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.547564] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.554043] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.561615] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.566676] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.572951] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.579571] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.585838] mdt_io01_068 S ffff8b5da986a080 0 119009 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.593203] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.595885] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.601074] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.607958] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.613749] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.620233] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.627823] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.632883] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.639158] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.645778] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.652053] mdt_io00_092 S ffff8b53668e9040 0 119011 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.659417] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.662048] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.667240] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.674120] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.679913] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.686394] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.693968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.699029] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.705303] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.711925] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.718199] mdt_io00_093 S ffff8b53668ea080 0 119014 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.725562] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.728211] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.733412] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.740294] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.746089] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.752574] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.760170] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.765228] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.771504] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:04 2019][2937706.778123] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.784395] mdt_io01_069 S ffff8b53668eb0c0 0 119015 2 0x00000080 [Fri Apr 12 11:13:04 2019][2937706.791751] Call Trace: [Fri Apr 12 11:13:04 2019][2937706.794383] [] schedule+0x29/0x70 [Fri Apr 12 11:13:04 2019][2937706.799575] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.806476] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:04 2019][2937706.812265] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.818748] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:04 2019][2937706.826320] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:04 2019][2937706.831379] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:04 2019][2937706.837658] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937706.844279] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937706.850570] mdt_io00_094 S ffff8b53668e8000 0 119016 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937706.857942] Call Trace: [Fri Apr 12 11:13:05 2019][2937706.860599] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937706.865802] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937706.872681] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937706.878486] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937706.884970] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937706.892548] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937706.897606] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937706.903903] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937706.910539] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937706.916844] mdt_rdpg02_006 S ffff8b65296a0000 0 119017 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937706.924214] Call Trace: [Fri Apr 12 11:13:05 2019][2937706.926848] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937706.932046] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937706.938949] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937706.944746] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937706.951235] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937706.958817] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937706.963879] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937706.970156] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937706.976777] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937706.983050] mdt_io00_095 S ffff8b5b7ffeb0c0 0 119018 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937706.990451] Call Trace: [Fri Apr 12 11:13:05 2019][2937706.993089] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937706.998282] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.005161] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.010955] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.017439] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.025028] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.030088] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.036361] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.042982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.049275] mdt_rdpg02_007 S ffff8b5b7ffec100 0 119019 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.056636] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.059268] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.064489] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.071369] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.077162] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.083660] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.091234] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.096294] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.102572] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.109191] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.115471] mdt_io01_070 S ffff8b569fae2080 0 119020 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.122870] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.125503] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.130693] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.137574] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.143364] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.149847] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.157421] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.162492] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.168786] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.175406] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.181680] mdt_io00_096 S ffff8b569fae30c0 0 119021 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.189050] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.191682] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.196877] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.203766] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.209558] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.216048] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.223632] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.228698] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.234973] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.241591] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.247868] mdt_rdpg01_006 S ffff8b569fae0000 0 119022 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.255231] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.257888] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.263140] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.270042] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.275837] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.282340] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.289951] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.295017] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.301293] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.307916] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.314186] mdt_rdpg02_008 S ffff8b569fae1040 0 119023 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.321559] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.324205] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.329402] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.336289] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.342105] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.348600] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.356189] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.361258] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.367535] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.374161] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.380445] mdt_rdpg01_007 S ffff8b569fae6180 0 119024 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.387807] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.390441] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.395631] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.402565] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.408357] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.414837] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.422410] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.427469] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.433757] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.440406] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.446677] mdt_io01_071 S ffff8b569fae4100 0 119025 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.454049] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.456681] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.461873] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.468776] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.474573] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.481073] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.488655] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.493713] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.499988] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.506618] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.512899] mdt_io00_097 S ffff8b7cf9b48000 0 119026 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.520263] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.522894] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.528092] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.535039] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.540862] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.547386] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.555016] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.560092] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.566422] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.573173] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.579444] mdt_rdpg02_009 S ffff8b7cf9b4c100 0 119027 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.587065] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.590429] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.595893] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.602778] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.608572] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.615223] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.622796] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.627856] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.634170] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.640799] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.647072] mdt_rdpg00_010 S ffff8b6da98d2080 0 119028 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.654434] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.657066] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.662260] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.669138] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.674932] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.681413] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.688985] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.694050] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.700341] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.707030] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.713436] mdt_io01_072 S ffff8b6c010a9040 0 119029 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.720806] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.723439] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.728632] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.735511] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.741306] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.747808] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.755380] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.760442] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.766721] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.773359] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.779630] mdt_rdpg00_011 S ffff8b6c010ae180 0 119030 2 0x00000080 [Fri Apr 12 11:13:05 2019][2937707.787011] Call Trace: [Fri Apr 12 11:13:05 2019][2937707.789646] [] schedule+0x29/0x70 [Fri Apr 12 11:13:05 2019][2937707.794835] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.801718] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:05 2019][2937707.807510] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.813999] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:05 2019][2937707.821573] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:05 2019][2937707.826630] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.832909] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:05 2019][2937707.839538] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:05 2019][2937707.845897] mdt_io00_098 S ffff8b502d710000 0 119031 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937707.853367] Call Trace: [Fri Apr 12 11:13:06 2019][2937707.856020] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937707.861247] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937707.868131] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937707.873923] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937707.880413] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937707.887988] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937707.893049] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937707.899324] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937707.905944] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937707.912217] mdt_rdpg00_012 S ffff8b502d712080 0 119032 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937707.919596] Call Trace: [Fri Apr 12 11:13:06 2019][2937707.922228] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937707.927421] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937707.934313] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937707.940126] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937707.946644] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937707.954223] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937707.959287] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937707.965565] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937707.972194] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937707.978475] mdt_io00_099 S ffff8b502d711040 0 119033 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937707.985855] Call Trace: [Fri Apr 12 11:13:06 2019][2937707.988488] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937707.993682] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.000579] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.006389] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.012997] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.020677] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.025804] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.032207] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.038928] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.045263] mdt_rdpg02_010 S ffff8b502d714100 0 119034 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.052643] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.055276] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.060478] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.067368] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.073158] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.079643] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.087220] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.092280] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.098559] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.105181] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.111450] mdt_io00_100 S ffff8b502d716180 0 119035 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.118823] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.121455] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.126648] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.133533] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.139351] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.145854] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.153456] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.158521] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.164799] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.171416] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.177694] mdt_rdpg01_008 S ffff8b502d715140 0 119036 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.185063] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.187696] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.192889] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.199772] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.205572] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.212058] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.219633] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.224693] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.230972] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.237589] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.243863] mdt_rdpg00_013 S ffff8b4e75690000 0 119037 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.251226] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.253860] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.259054] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.265943] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.271734] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.278231] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.286020] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.291131] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.297764] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.304468] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.310744] mdt_rdpg01_009 S ffff8b4e75694100 0 119038 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.318110] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.320750] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.325946] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.332832] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.338626] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.345132] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.352714] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.357777] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.364061] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.370679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.376960] mdt_io01_073 S ffff8b4e756930c0 0 119039 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.384334] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.386972] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.392179] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.399072] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.404863] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.411343] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.418936] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.423996] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.430270] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.436891] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.443167] mdt_rdpg00_014 S ffff8b7903348000 0 119040 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.450538] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.453170] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.458363] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.465245] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.471039] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.477525] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.485099] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.490159] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.496444] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.503074] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.509354] mdt_rdpg00_015 S ffff8b7903349040 0 119041 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.516726] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.519360] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.524548] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.531433] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.537225] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.543712] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.551300] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.556368] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.562645] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.569271] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.575543] mdt_rdpg01_010 S ffff8b631a72a100 0 119042 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.582914] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.585548] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.590740] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.597622] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.603414] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.609896] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.617467] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.622529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.628815] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.635435] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.641715] mdt_rdpg00_016 S ffff8b4e75696180 0 119043 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.649086] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.651728] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.656918] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.663801] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.669591] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.676091] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.683665] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.688723] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.694991] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.701611] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.707890] mdt_io01_074 S ffff8b4e75692080 0 119044 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.715258] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.717891] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.723081] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.729971] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.735772] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.742254] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.749836] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.754902] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.761183] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.767803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.774076] mdt_rdpg00_017 S ffff8b6cf5322080 0 119045 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.781447] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.784080] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.789277] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.796203] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.801995] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.808475] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.816054] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:06 2019][2937708.821122] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.827397] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:06 2019][2937708.834018] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:06 2019][2937708.840291] mdt_rdpg02_011 S ffff8b6cf5326180 0 119046 2 0x00000080 [Fri Apr 12 11:13:06 2019][2937708.847653] Call Trace: [Fri Apr 12 11:13:06 2019][2937708.850286] [] schedule+0x29/0x70 [Fri Apr 12 11:13:06 2019][2937708.855475] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:06 2019][2937708.862360] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:06 2019][2937708.868151] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937708.874642] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937708.882235] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937708.887292] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937708.893571] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937708.900199] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937708.906480] mdt_io00_101 S ffff8b6cf5325140 0 119047 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937708.913861] Call Trace: [Fri Apr 12 11:13:07 2019][2937708.916492] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937708.921682] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937708.928563] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937708.934357] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937708.940865] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937708.948448] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937708.953509] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937708.959781] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937708.966424] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937708.972694] mdt_rdpg02_012 S ffff8b6cf5320000 0 119048 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937708.980056] Call Trace: [Fri Apr 12 11:13:07 2019][2937708.982692] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937708.987885] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937708.994765] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.000561] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.007372] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.015029] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.020087] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.026364] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.032982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.039258] mdt_io01_075 S ffff8b6cf5321040 0 119049 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.046636] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.049270] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.054467] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.061351] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.067140] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.073624] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.081197] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.086257] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.092535] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.099175] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.105448] mdt_io01_076 S ffff8b6cf5324100 0 119050 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.112817] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.115452] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.120645] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.127532] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.133325] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.139822] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.147404] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.152473] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.158750] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.165370] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.171650] mdt_rdpg01_011 S ffff8b8ab8676180 0 119051 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.179023] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.181656] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.186857] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.193738] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.199528] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.206010] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.213584] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.218642] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.224922] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.231540] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.237813] mdt_io01_077 S ffff8b8ab8672080 0 119052 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.245185] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.247821] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.253018] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.259901] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.265690] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.272181] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.279755] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.284813] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.291093] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.297722] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.303994] mdt_io00_102 S ffff8b8ab86730c0 0 119053 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.311364] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.313996] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.319207] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.326089] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.331885] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.338397] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.345971] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.351029] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.357308] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.363928] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.370200] mdt_rdpg02_013 S ffff8b8ab8674100 0 119054 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.377572] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.380206] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.385397] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.392276] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.398065] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.404550] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.412120] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.417183] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.423461] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.430080] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.436352] mdt_io01_078 S ffff8b8ab8675140 0 119055 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.443708] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.446337] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.451527] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.458423] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.464212] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.470695] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.478268] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.483330] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.489615] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.496246] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.502523] mdt_rdpg02_014 S ffff8b8ab8670000 0 119056 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.509906] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.512540] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.517733] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.524619] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.530418] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.536908] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.544485] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.549545] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.555833] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.562451] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.568723] mdt_rdpg01_012 S ffff8b8ab8671040 0 119057 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.576087] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.578719] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.583912] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.590793] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.596587] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.603080] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.610656] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.615717] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.622011] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.628631] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.634904] mdt_rdpg01_013 S ffff8b72a0efb0c0 0 119058 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.642267] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.644916] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.650105] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.656990] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.662783] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.669283] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.676860] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.681924] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.688208] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.694856] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.701124] mdt_io00_103 S ffff8b72a0efc100 0 119059 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.708481] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.711112] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.716306] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.723187] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.728987] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.735472] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.743048] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.748108] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.754376] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.761019] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.767291] mdt_io01_079 S ffff8b72a0ef9040 0 119060 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.774670] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.777310] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.782504] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.789394] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:07 2019][2937709.795183] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.801665] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:07 2019][2937709.809254] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:07 2019][2937709.814318] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.820630] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:07 2019][2937709.827260] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:07 2019][2937709.833541] mdt_rdpg01_014 S ffff8b72a0ef8000 0 119061 2 0x00000080 [Fri Apr 12 11:13:07 2019][2937709.840912] Call Trace: [Fri Apr 12 11:13:07 2019][2937709.843545] [] schedule+0x29/0x70 [Fri Apr 12 11:13:07 2019][2937709.848747] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937709.855634] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937709.861419] [] ? class_conn2export+0x4a/0x2c0 [obdclass] [Fri Apr 12 11:13:08 2019][2937709.872541] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937709.879393] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937709.886965] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937709.892022] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937709.898309] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937709.904934] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937709.911203] mdt_rdpg02_015 S ffff8b72a0efa080 0 119062 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937709.918566] Call Trace: [Fri Apr 12 11:13:08 2019][2937709.921199] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937709.926391] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937709.933554] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937709.940777] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937709.947278] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937709.954861] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937709.959923] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937709.966195] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937709.972819] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937709.979091] mdt_io00_104 S ffff8b72a0efd140 0 119063 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937709.986463] Call Trace: [Fri Apr 12 11:13:08 2019][2937709.989097] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937709.994289] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.001175] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.006968] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.013448] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.021025] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.026083] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.032360] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.038982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.045255] mdt_io00_105 S ffff8b72a0efe180 0 119064 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.052615] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.055249] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.060449] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.067372] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.073170] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.079658] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.087240] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.092308] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.098594] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.105214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.111494] mdt_io01_080 S ffff8b5cfb304100 0 119065 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.118892] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.121525] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.126717] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.133597] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.139389] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.145872] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.153443] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.158507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.164781] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.171402] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.177674] mdt_rdpg01_015 S ffff8b5cfb301040 0 119066 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.185063] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.187710] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.192909] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.199802] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:13:08 2019][2937710.206791] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.212590] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.219088] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.226699] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.231758] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.238034] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.244655] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.250943] mdt_rdpg02_016 S ffff8b57d5ee8000 0 119067 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.258335] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.260968] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.266164] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.273048] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.278844] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.285333] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.292909] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.297971] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.304247] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.310869] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.317160] mdt_rdpg01_016 S ffff8b57d5ee9040 0 119068 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.324522] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.327153] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.332342] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.339226] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.345017] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.351501] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.359073] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.364133] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.370422] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.377038] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.383312] mdt_rdpg02_017 S ffff8b75446fe180 0 119069 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.390676] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.393306] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.398495] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.405373] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.411164] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.417645] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.425220] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.430278] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.436555] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.443176] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.449448] mdt_io01_081 S ffff8b75446f9040 0 119070 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.456802] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.459435] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.464621] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.471501] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.477295] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.483783] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.491382] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.496444] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.502719] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.509348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.515622] mdt_rdpg01_017 S ffff8b6debbc0000 0 119071 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.523000] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.525635] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.530827] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.537707] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.543500] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.549996] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.557576] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.562641] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.568919] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.575544] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.581829] mdt_io01_082 S ffff8b6debbc1040 0 119072 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.589207] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.591840] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.597033] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.603913] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.609707] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.616193] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.623773] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.628836] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.635137] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.641812] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.648095] mdt_rdpg00_018 S ffff8b6debbc2080 0 119073 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.655458] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.658091] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.663287] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.670202] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.676015] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.682504] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.690081] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.695138] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.701414] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.708037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.714318] mdt_rdpg02_018 S ffff8b6debbc30c0 0 119074 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.721680] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.724314] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.729503] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.736396] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.742194] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.748689] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.756278] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.761339] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.767624] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.774261] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.780557] mdt_io01_083 S ffff8b6debbc4100 0 119075 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.787913] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.790545] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.795739] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.802621] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:08 2019][2937710.808417] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.814905] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:08 2019][2937710.822483] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:08 2019][2937710.827542] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.833818] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:08 2019][2937710.840440] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:08 2019][2937710.846714] mdt_rdpg01_018 S ffff8b6debbc5140 0 119076 2 0x00000080 [Fri Apr 12 11:13:08 2019][2937710.854087] Call Trace: [Fri Apr 12 11:13:08 2019][2937710.856746] [] schedule+0x29/0x70 [Fri Apr 12 11:13:08 2019][2937710.861938] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937710.868827] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937710.874638] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937710.881127] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937710.888716] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937710.893775] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937710.900089] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937710.906709] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937710.912990] mdt_rdpg00_019 S ffff8b6debbc6180 0 119077 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937710.920362] Call Trace: [Fri Apr 12 11:13:09 2019][2937710.923011] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937710.928206] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937710.935103] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:13:09 2019][2937710.942079] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937710.947881] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937710.954370] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937710.961962] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937710.967020] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937710.973298] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937710.979933] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937710.986209] mdt_io01_084 S ffff8b758de90000 0 119078 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937710.993604] Call Trace: [Fri Apr 12 11:13:09 2019][2937710.996269] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.001475] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.008383] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.014192] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.022880] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.030574] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.035637] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.041915] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.048561] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.054841] mdt_rdpg02_019 S ffff8b758de91040 0 119079 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.062240] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.064890] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.070091] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.076990] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.082793] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.089306] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.096912] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.101982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.108261] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.114911] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.121193] mdt_rdpg02_020 S ffff8b78aca25140 0 119080 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.128555] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.131190] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.136413] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.143473] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.153396] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.160309] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.167902] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.172964] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.179471] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.186356] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.192724] mdt_rdpg02_021 S ffff8b7ba5afa080 0 119081 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.200094] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.202746] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.207948] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.214864] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.220676] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.227180] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.234794] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.239851] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.246127] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.252746] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.259024] mdt_rdpg00_020 S ffff8b7ba5afb0c0 0 119082 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.266387] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.269038] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.274230] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.281139] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.286938] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.293438] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.301031] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.306113] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.312413] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.319037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.325340] mdt_rdpg01_019 S ffff8b7ba5afc100 0 119083 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.332703] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.335336] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.340574] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.347479] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.353275] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.359770] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.367346] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.372402] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.378681] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.385982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.392406] mdt_rdpg01_020 S ffff8b8cbaaa30c0 0 119085 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.399778] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.402409] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.407610] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.414501] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.420315] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.426834] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.434410] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.439468] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.445754] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.452377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.458645] mdt_rdpg02_022 S ffff8b8cbaaa0000 0 119086 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.466008] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.468672] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.473870] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.480749] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.486542] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.493039] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.500618] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.505685] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.512030] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.518650] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.524954] mdt_rdpg01_021 S ffff8b8cbaaa4100 0 119087 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.532356] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.534998] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.540202] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.547086] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.552887] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.559393] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.566970] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.572028] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.578325] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.584965] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.591251] mdt_rdpg02_023 S ffff8b8cbaaa2080 0 119088 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.598614] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.601255] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.606448] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.613328] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.619123] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.625614] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.633190] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.638252] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.644537] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.651160] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.657454] mdt_rdpg00_021 S ffff8b8cbaaa6180 0 119089 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.664828] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.667461] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.672657] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.679564] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.685370] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.691864] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.699440] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.704505] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.710829] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.717463] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.723743] mdt_io00_106 S ffff8b7a88390000 0 119090 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.731113] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.733765] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.738975] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.745870] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.751672] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.758169] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.765745] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.770908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.777329] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:09 2019][2937711.784050] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:09 2019][2937711.790382] mdt_rdpg02_024 S ffff8b7a883930c0 0 119091 2 0x00000080 [Fri Apr 12 11:13:09 2019][2937711.797794] Call Trace: [Fri Apr 12 11:13:09 2019][2937711.800427] [] schedule+0x29/0x70 [Fri Apr 12 11:13:09 2019][2937711.805632] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.812519] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:13:09 2019][2937711.819542] [] ? sptlrpc_wireflavor2policy+0x5d/0x1c0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.827219] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:09 2019][2937711.833076] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.839579] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:09 2019][2937711.847168] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:09 2019][2937711.852259] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937711.858541] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937711.865173] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937711.871449] mdt_rdpg01_022 S ffff8b7a88395140 0 119092 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937711.878838] Call Trace: [Fri Apr 12 11:13:10 2019][2937711.881474] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937711.886678] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937711.893561] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937711.899373] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937711.905874] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937711.913471] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937711.918530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937711.924923] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937711.931873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937711.938221] mdt_io01_085 S ffff8b7a88391040 0 119093 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937711.945592] Call Trace: [Fri Apr 12 11:13:10 2019][2937711.948529] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937711.953872] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937711.960831] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937711.966641] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937711.973147] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937711.980739] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937711.985805] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937711.992078] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937711.998696] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.004979] mdt_rdpg01_023 S ffff8b7a88394100 0 119094 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.012361] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.015001] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.020195] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.027071] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.032868] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.039355] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.046932] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.052002] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.058281] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.064906] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.071240] mdt_rdpg02_025 S ffff8b7a88396180 0 119095 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.078740] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.081376] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.086613] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.093499] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.099301] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.105854] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.113452] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.118517] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.124810] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.131452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.137739] mdt_rdpg02_026 S ffff8b7a88392080 0 119096 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.145121] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.147896] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.153138] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.160025] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.165835] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.172332] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.179964] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.185027] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.191311] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.197946] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.204222] mdt_rdpg01_024 S ffff8b5c6b398000 0 119097 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.211583] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.214215] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.219415] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.226304] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.232100] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.238581] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.246153] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.251233] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.257659] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.264284] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.270556] mdt_rdpg01_025 S ffff8b5c6b399040 0 119098 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.277930] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.280573] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.285768] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.292694] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.298523] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.305010] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.312618] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.317684] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.323968] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.330631] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.336965] mdt_rdpg00_022 S ffff8b5c6b39c100 0 119099 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.344345] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.346988] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.352192] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.359095] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.364899] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.371405] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.378995] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.384075] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.390363] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.396982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.403261] mdt_rdpg01_026 S ffff8b88d036a080 0 119100 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.410629] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.413274] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.418481] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.425390] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.431210] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.437702] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.445292] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.450386] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:13:10 2019][2937712.456859] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.463158] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.470092] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.476492] mdt_rdpg00_023 S ffff8b88d0369040 0 119101 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.483854] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.486491] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.491829] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.498735] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.504545] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.511068] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.518661] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.523755] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.530048] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.536678] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.542949] mdt_rdpg01_027 S ffff8b88d036b0c0 0 119102 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.550322] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.552966] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.558184] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.565110] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.570912] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.577412] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.585052] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.590119] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.596411] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.603030] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.609470] mdt_rdpg02_027 S ffff8b5cf8e7b0c0 0 119103 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.616850] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.619512] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.624728] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.631711] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.637518] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.644176] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.651789] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.657025] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.663317] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.670112] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.676386] mdt_rdpg00_024 S ffff8b5cf8e79040 0 119104 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.683757] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.686426] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.691647] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.698541] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.704356] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.710849] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.718426] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.723503] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.729774] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.736394] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.742668] mdt_io01_086 S ffff8b5c6b39b0c0 0 119105 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.750031] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.752666] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.757858] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.764738] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.770573] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.777073] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.784771] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.790163] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.796796] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:10 2019][2937712.803418] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.809693] mdt_rdpg01_028 S ffff8b6575628000 0 119106 2 0x00000080 [Fri Apr 12 11:13:10 2019][2937712.817434] Call Trace: [Fri Apr 12 11:13:10 2019][2937712.820067] [] schedule+0x29/0x70 [Fri Apr 12 11:13:10 2019][2937712.825270] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.832177] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:10 2019][2937712.837970] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.844468] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:10 2019][2937712.852046] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:10 2019][2937712.857128] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:10 2019][2937712.863409] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937712.870030] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937712.876312] mdt_rdpg00_025 S ffff8b59d874a400 0 119107 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937712.883673] Call Trace: [Fri Apr 12 11:13:11 2019][2937712.886306] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937712.892255] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937712.899148] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937712.904959] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937712.911634] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937712.919236] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937712.924466] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937712.930741] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937712.937533] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937712.943811] mdt_rdpg01_029 S ffff8b5cf8e7a080 0 119108 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937712.951192] Call Trace: [Fri Apr 12 11:13:11 2019][2937712.953831] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937712.960129] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937712.967033] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937712.972832] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937712.979327] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937712.986900] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937712.991962] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937712.998237] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.004864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.011161] mdt_rdpg00_026 S ffff8b5cf8e7d140 0 119109 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.018530] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.021166] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.026366] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.033253] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.039054] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.045537] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.053107] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.058167] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.064443] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.071065] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.077336] mdt_rdpg01_030 S ffff8b5cf8e7e180 0 119110 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.084701] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.087337] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.092523] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.099426] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.105219] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.111708] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.119279] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.124345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.130619] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.137679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.142337] Lustre: 118314:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply [Fri Apr 12 11:13:11 2019][2937713.142337] req@ffff8b59d874d700 x1629289855921312/t0(0) o36->5b697899-e91c-7cc6-c44a-da5d20d26539@10.9.101.58@o2ib4:16/0 lens 528/2888 e 0 to 0 dl 1555092796 ref 2 fl Interpret:/0/0 rc 0/0 [Fri Apr 12 11:13:11 2019][2937713.142340] Lustre: 118314:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 29 previous similar messages [Fri Apr 12 11:13:11 2019][2937713.183640] mdt_io01_087 S ffff8b6cfb77a080 0 119111 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.191000] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.193639] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.198841] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.205723] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.211517] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.218000] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.225579] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.230638] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.236916] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.243535] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.249807] mdt_rdpg01_031 S ffff8b6cfb77d140 0 119112 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.257164] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.259796] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.264984] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.271879] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.277714] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.284227] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.292112] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.297292] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.303648] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.310278] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.316843] mdt_rdpg01_032 S ffff8b6cfb778000 0 119113 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.324276] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.326926] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.332136] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.339041] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.344903] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.351394] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.359392] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.364836] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.371124] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.377759] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.384032] mdt_io01_088 S ffff8b6cfb77b0c0 0 119114 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.391413] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.394049] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.399244] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.406130] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.411921] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.418562] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.426167] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.431248] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.437999] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.444837] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.451107] mdt_io00_107 S ffff8b740ee54100 0 119115 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.458477] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.461112] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.466308] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.473208] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.479086] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.485598] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.493188] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.498250] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.504545] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.511171] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.517443] mdt_rdpg03_029 S ffff8b740ee56180 0 119116 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.524805] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.527439] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.532645] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.539532] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.545420] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.551920] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.559506] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.564946] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.571433] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.578053] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.584498] mdt_io01_089 S ffff8b740ee51040 0 119117 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.591914] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.594550] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.599750] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.606634] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.612429] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.618914] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.626492] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.631550] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.637828] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.644453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.650730] mdt_io01_090 S ffff8b5cf660e180 0 119118 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.658130] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.660770] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.665961] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.672860] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.678713] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.685211] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.692823] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.697908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.704214] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.710840] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.717111] mdt_rdpg02_028 S ffff8b7a7c29a080 0 119119 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.724482] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.727114] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.732318] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.739224] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.745022] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.751501] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.759077] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.764135] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.770422] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.777042] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.783315] mdt_io01_091 S ffff8b7a7c29d140 0 119120 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.790679] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.793310] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.798501] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.805383] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:11 2019][2937713.811177] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.817659] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.825289] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:11 2019][2937713.830366] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.836668] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:11 2019][2937713.843293] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:11 2019][2937713.849593] mdt_rdpg02_029 S ffff8b7a7c298000 0 119121 2 0x00000080 [Fri Apr 12 11:13:11 2019][2937713.856987] Call Trace: [Fri Apr 12 11:13:11 2019][2937713.859621] [] schedule+0x29/0x70 [Fri Apr 12 11:13:11 2019][2937713.864833] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:11 2019][2937713.871727] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937713.877544] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937713.884047] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937713.891630] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937713.896689] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937713.902964] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937713.909594] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937713.916001] mdt_io01_092 S ffff8b7a7c29c100 0 119122 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937713.923370] Call Trace: [Fri Apr 12 11:13:12 2019][2937713.926004] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937713.931248] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937713.938170] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937713.943976] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937713.950460] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937713.958046] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937713.963101] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937713.969380] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937713.976013] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937713.982290] mdt_rdpg02_030 S ffff8b7ba5afd140 0 119123 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937713.990326] Call Trace: [Fri Apr 12 11:13:12 2019][2937713.993096] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937713.998293] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.005172] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.010963] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.017444] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.025018] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.030076] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.036345] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.042966] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.049239] mdt_rdpg03_030 S ffff8b7ba5afe180 0 119124 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.056594] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.059228] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.064416] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.071297] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.077099] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.083586] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.091166] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.096241] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.102518] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.109138] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.115412] mdt_io01_093 S ffff8b7a70638000 0 119125 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.122775] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.125406] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.130599] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.137478] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:13:12 2019][2937714.144449] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.150243] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.156735] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.164312] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.169371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.175647] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.182268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.188544] mdt_rdpg00_027 S ffff8b7a70639040 0 119126 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.195905] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.198536] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.203725] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.210603] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.216397] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.222931] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.230584] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.235641] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.241926] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.248546] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.254817] mdt_io00_108 S ffff8b7a7063a080 0 119127 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.262175] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.264837] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.270045] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.276925] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.282739] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.289269] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.296928] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.301987] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.308275] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.314911] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.321192] mdt_rdpg00_028 S ffff8b7a7063b0c0 0 119128 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.328942] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.331581] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.336838] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.343723] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.349540] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.356090] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.363689] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.368775] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.375060] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.381679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.387950] mdt_rdpg03_031 S ffff8b7a7063c100 0 119129 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.395335] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.397972] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.403165] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.410045] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.415866] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.422365] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.430170] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.435278] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.441563] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.448336] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.454677] mdt_io01_094 S ffff8b7a7063d140 0 119130 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.462049] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.464684] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.469901] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.476882] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.482695] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.489227] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.496811] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.501869] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.508143] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.514764] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.521038] mdt_rdpg02_031 S ffff8b7a7063e180 0 119131 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.528401] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.531035] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.536227] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.543104] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.548899] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.555389] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.562963] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.568020] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.574289] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.580907] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.587177] mdt_rdpg01_033 S ffff8b7985d30000 0 119132 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.594564] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.597198] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.602467] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.609395] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.615188] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.621676] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.629245] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.634308] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.640583] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.647205] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.653475] mdt_io00_109 S ffff8b7985d31040 0 119133 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.660831] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.663464] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.668651] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.675528] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.681320] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.687802] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.695374] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.700434] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.706709] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.713331] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.719604] mdt_rdpg03_032 S ffff8b7985d32080 0 119134 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.726961] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.729591] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.734795] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.741684] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.747478] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.753993] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.761598] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.766663] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.772964] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.779584] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.785856] mdt_io00_110 S ffff8b7985d330c0 0 119135 2 0x00000080 [Fri Apr 12 11:13:12 2019][2937714.793230] Call Trace: [Fri Apr 12 11:13:12 2019][2937714.795873] [] schedule+0x29/0x70 [Fri Apr 12 11:13:12 2019][2937714.801408] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.808315] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:12 2019][2937714.814117] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.820612] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:12 2019][2937714.828191] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:12 2019][2937714.833257] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.839532] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:12 2019][2937714.846163] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:12 2019][2937714.852436] mdt_rdpg02_032 S ffff8b7985d34100 0 119136 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937714.859798] Call Trace: [Fri Apr 12 11:13:13 2019][2937714.862431] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937714.867623] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937714.874503] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937714.880298] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937714.886781] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937714.894358] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937714.899417] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937714.905694] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937714.912311] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937714.918577] mdt_rdpg00_029 S ffff8b7985d35140 0 119137 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937714.925935] Call Trace: [Fri Apr 12 11:13:13 2019][2937714.928569] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937714.933762] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937714.940643] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937714.946441] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937714.952942] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937714.960533] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937714.965602] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937714.971876] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937714.978494] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937714.984768] mdt_io01_095 S ffff8b7985d36180 0 119138 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937714.992131] Call Trace: [Fri Apr 12 11:13:13 2019][2937714.994764] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937714.999958] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.006838] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.012629] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.019111] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.026691] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.031756] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.038033] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.044728] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.051001] mdt_rdpg02_033 S ffff8b758de92080 0 119139 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.058381] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.061017] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.066236] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.073122] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.078924] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.085413] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.092995] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.098066] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.104339] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.111022] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.117323] mdt_rdpg01_034 S ffff8b758de930c0 0 119140 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.124745] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.127388] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.132619] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.139499] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.145368] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.151852] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.159427] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.164487] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.170879] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.177541] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.183811] mdt_rdpg02_034 S ffff8b758de94100 0 119141 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.191218] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.193856] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.199065] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.205951] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:13:13 2019][2937715.212929] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.218727] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.225220] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.232790] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.237849] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.244126] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.250745] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.257024] mdt_rdpg01_035 S ffff8b758de95140 0 119142 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.264383] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.267015] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.272209] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.279091] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.284931] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.291431] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.299009] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.304062] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.310335] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.316951] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.323506] mdt_rdpg00_030 S ffff8b758de96180 0 119143 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.331111] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.333763] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.338959] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.345863] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.351682] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.358167] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.365743] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.370824] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.377104] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.383723] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.389998] mdt_io00_111 S ffff8b7ca6fc0000 0 119144 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.397385] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.400041] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.405317] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.412209] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.418013] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.424521] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.432110] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.437356] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.443762] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.450393] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.456662] mdt_io01_096 S ffff8b7ca6fc1040 0 119145 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.464026] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.466657] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.471896] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.479002] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.484910] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.491445] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.499030] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.504090] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.510371] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.516994] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.523269] mdt_io01_097 S ffff8b7ca6fc2080 0 119146 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.530650] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.533289] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.538496] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.545431] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.551225] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.557713] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.565286] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.570343] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.576612] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.583234] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.589506] mdt_rdpg02_035 S ffff8b7ca6fc30c0 0 119147 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.596862] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.599493] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.604685] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.611564] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.617363] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.623870] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.631459] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.636521] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.642797] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.649415] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.655709] mdt_rdpg00_031 S ffff8b7ca6fc4100 0 119148 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.663077] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.665712] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.670915] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.677821] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.683625] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.690119] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.697700] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.702801] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.709090] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.716120] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.722511] mdt_rdpg00_032 S ffff8b5736bc8000 0 119149 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.729876] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.732520] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.737755] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.744651] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.750453] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.756946] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.764518] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.769578] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.775860] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.782494] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.788770] mdt_io01_098 S ffff8b5736bc9040 0 119150 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.796456] Call Trace: [Fri Apr 12 11:13:13 2019][2937715.799269] [] schedule+0x29/0x70 [Fri Apr 12 11:13:13 2019][2937715.804479] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.811383] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:13 2019][2937715.817189] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.823733] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:13 2019][2937715.831329] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:13 2019][2937715.836395] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.842679] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:13 2019][2937715.849297] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:13 2019][2937715.855569] mdt_rdpg03_033 S ffff8b57d5eea080 0 119151 2 0x00000080 [Fri Apr 12 11:13:13 2019][2937715.862929] Call Trace: [Fri Apr 12 11:13:14 2019][2937715.865569] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937715.870761] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937715.877644] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937715.883461] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937715.889952] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937715.897545] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937715.902620] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937715.908894] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937715.915553] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937715.921996] mdt_rdpg01_036 R running task 0 119152 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937715.929447] Call Trace: [Fri Apr 12 11:13:14 2019][2937715.932106] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937715.937304] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937715.944265] [] ? sptlrpc_wireflavor2policy+0x5d/0x1c0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937715.951937] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937715.957770] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937715.964279] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937715.971867] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937715.976930] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937715.983205] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937715.989824] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937715.996098] ldlm_cn03_005 S ffff8b5b31038000 0 119153 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.003722] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.006366] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.011642] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.018842] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.024707] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.031211] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.038786] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.043848] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.050159] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.056786] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.063067] mdt_rdpg00_033 S ffff8b5b31039040 0 119154 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.070429] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.073064] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.079849] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.086746] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.092561] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.099125] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.106709] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.111785] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.118070] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.124700] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.130988] mdt_rdpg02_036 S ffff8b5b3103a080 0 119155 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.138371] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.141008] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.146224] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.153120] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.158913] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.165401] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.172980] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.178054] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.184325] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.190956] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.197233] mdt_rdpg00_034 S ffff8b57d5eec100 0 119156 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.204611] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.207250] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.212478] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.219363] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.225172] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.231676] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.239293] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.244353] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.250647] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.257295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.263568] mdt_rdpg00_035 S ffff8b53668ee180 0 119158 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.270931] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.273637] [] ? tgt_request_handle+0x697/0x1580 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.280868] [] ? schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.286241] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.293296] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.299089] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.305747] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.313321] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.318560] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.324870] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.331724] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.338008] mdt_rdpg02_037 S ffff8b53668ed140 0 119159 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.345386] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.348021] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.353211] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.360092] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.365893] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.372429] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.380017] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.385086] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.391372] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.397996] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.404276] mdt_rdpg02_038 S ffff8b7a7c29b0c0 0 119160 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.411644] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.414278] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.419542] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.426437] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.432229] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.438715] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.446288] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.451348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.457648] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.464262] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.470538] mdt_rdpg00_036 R running task 0 119161 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.477901] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.480593] [] ? tgt_lookup_reply_by_xid+0x3c/0x60 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.487997] [] ? dequeue_task_fair+0x41e/0x660 [Fri Apr 12 11:13:14 2019][2937716.494267] [] ? sched_clock_cpu+0x85/0xc0 [Fri Apr 12 11:13:14 2019][2937716.500198] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:13:14 2019][2937716.505951] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:13:14 2019][2937716.512571] [] ? probe_sched_switch+0x2d/0xa0 [Fri Apr 12 11:13:14 2019][2937716.518762] [] ? schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.524125] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.531184] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.536972] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.543628] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.551201] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.556433] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.562711] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.569504] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.575779] mdt_io01_099 S ffff8b7cf96e8000 0 119162 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.583161] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.585791] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.590982] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.597866] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.603656] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.610142] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.617711] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.622771] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.629048] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.635669] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.641956] mdt_rdpg02_039 S ffff8b7cf96e9040 0 119163 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.649313] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.651944] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.657136] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.664020] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.669813] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.676290] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.683866] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.688927] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.695203] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.701839] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.708114] mdt_rdpg00_037 S ffff8b7cf96ea080 0 119164 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.715476] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.718109] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.723298] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.730182] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.735975] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.742454] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.750028] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.755089] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.761365] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.767984] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.774259] mdt_rdpg03_034 S ffff8b7cf96eb0c0 0 119165 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.781622] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.784254] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.789445] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.796327] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.802118] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.808600] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.816176] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:14 2019][2937716.821235] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.827510] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:14 2019][2937716.834130] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:14 2019][2937716.840403] mdt_rdpg02_040 S ffff8b7cf96ec100 0 119166 2 0x00000080 [Fri Apr 12 11:13:14 2019][2937716.847760] Call Trace: [Fri Apr 12 11:13:14 2019][2937716.850399] [] schedule+0x29/0x70 [Fri Apr 12 11:13:14 2019][2937716.855603] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:14 2019][2937716.862540] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:14 2019][2937716.868344] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937716.874831] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937716.882410] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937716.887513] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937716.893786] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937716.900406] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937716.906683] mdt_rdpg02_041 S ffff8b7cf96ed140 0 119167 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937716.914043] Call Trace: [Fri Apr 12 11:13:15 2019][2937716.916679] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937716.921890] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937716.928783] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937716.934621] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937716.941126] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937716.948710] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937716.953855] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937716.960133] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937716.966753] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937716.973054] mdt_rdpg02_042 S ffff8b7cf96ee180 0 119168 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937716.980424] Call Trace: [Fri Apr 12 11:13:15 2019][2937716.983057] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937716.988252] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937716.995142] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.000990] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.007477] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.015064] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.020126] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.026439] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.033067] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.039355] mdt_io01_100 S ffff8b7cf7ed8000 0 119169 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.046840] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.049480] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.054682] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.061568] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.067378] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.073867] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.081454] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.086522] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.092797] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.099428] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.105700] mdt_rdpg01_037 D ffff8b7ca6fc5140 0 119170 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.113072] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.115712] [] schedule_preempt_disabled+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.122333] [] __mutex_lock_slowpath+0xc7/0x1d0 [Fri Apr 12 11:13:15 2019][2937717.128712] [] mutex_lock+0x1f/0x2f [Fri Apr 12 11:13:15 2019][2937717.134113] [] tgt_handle_received_xid+0x56/0xe0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.141413] [] tgt_request_handle+0xca8/0x1580 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.148511] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.156268] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:13:15 2019][2937717.163542] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.171422] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.178404] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:15 2019][2937717.184856] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:15 2019][2937717.190915] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.197436] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.205022] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.210082] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.216372] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.223018] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.229307] mdt_rdpg02_043 S ffff8b7ca6fc6180 0 119171 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.236694] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.239329] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.244530] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.251456] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.257260] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.263762] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.271352] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.276423] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.283175] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.290034] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.296321] mdt_rdpg01_038 S ffff8b76887b8000 0 119172 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.303697] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.306331] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.311528] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.318428] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.324226] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.330749] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.338337] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.343404] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.349686] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.356322] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.362593] mdt_rdpg03_035 S ffff8b76887b9040 0 119173 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.369965] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.372597] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.377797] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.384693] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.390497] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.396999] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.404670] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.409762] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.416035] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.422655] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.428948] mdt_rdpg01_039 S ffff8b75e9708000 0 119174 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.436379] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.439014] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.444277] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.451183] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.456982] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.463475] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.471073] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.476128] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.482408] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.489037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.495339] mdt_io01_101 S ffff8b75e9709040 0 119175 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.502733] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.505395] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.510596] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.517482] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.523285] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.529790] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.537362] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.542422] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.548711] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.555345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.561618] mdt_rdpg03_036 S ffff8b75e970a080 0 119176 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.568992] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.571642] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.576837] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.583737] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.589557] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.596041] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.603623] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.608682] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.614977] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.621625] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.627895] mdt_rdpg01_040 S ffff8b75e970b0c0 0 119177 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.635266] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.637909] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.643133] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.650018] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.655814] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.662295] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.669874] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.674932] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.681217] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.687842] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.694120] mdt_rdpg00_038 S ffff8b7cba6c8000 0 119178 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.701490] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.704124] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.709316] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.716200] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.724500] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.731047] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.738625] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.743684] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.749962] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.756600] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.762872] mdt_rdpg01_041 R running task 0 119179 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.770257] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.772896] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.778088] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.784975] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:15 2019][2937717.790782] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.797305] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.804902] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:15 2019][2937717.809972] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.816249] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:15 2019][2937717.822885] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:15 2019][2937717.829161] mdt_rdpg01_042 S ffff8b7cba6ca080 0 119180 2 0x00000080 [Fri Apr 12 11:13:15 2019][2937717.836528] Call Trace: [Fri Apr 12 11:13:15 2019][2937717.839162] [] schedule+0x29/0x70 [Fri Apr 12 11:13:15 2019][2937717.844366] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:15 2019][2937717.851248] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937717.857051] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937717.863547] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937717.871127] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937717.876184] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937717.882480] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937717.889106] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937717.895381] mdt_rdpg00_039 S ffff8b7cba6cb0c0 0 119181 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937717.902752] Call Trace: [Fri Apr 12 11:13:16 2019][2937717.905384] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937717.910594] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937717.917484] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937717.923284] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937717.929787] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937717.937391] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937717.942451] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937717.948730] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937717.955346] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937717.961630] mdt_io01_102 S ffff8b7cba6cc100 0 119182 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937717.969201] Call Trace: [Fri Apr 12 11:13:16 2019][2937717.971934] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937717.977132] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937717.984020] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937717.989831] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937717.996332] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.003910] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.008966] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.015243] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.021864] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.028139] mdt_rdpg01_043 S ffff8b7cba6cd140 0 119183 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.035503] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.038134] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.043320] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.050202] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.056048] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.062534] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.070126] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.075187] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.081472] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.088108] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.094379] mdt_rdpg02_044 S ffff8b7cba6ce180 0 119184 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.101806] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.104448] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.109640] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.116526] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.122327] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.128811] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.136389] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.141449] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.147724] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.154348] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.160619] mdt_rdpg02_045 S ffff8b792a378000 0 119185 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.167994] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.170637] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.175876] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.182798] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.188614] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.195102] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.202682] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.207747] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.214024] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.220670] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.226967] mdt_rdpg00_040 R running task 0 119186 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.234624] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.237449] [] ? tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.244693] [] ? schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.250057] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.257123] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.262917] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.269593] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.277218] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.282451] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.288738] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.295601] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.301873] mdt_rdpg01_044 S ffff8b792a37a080 0 119187 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.309247] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.311905] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.317127] [] ptlrpc_wait_event+0x40/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.323924] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.329718] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.336236] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.343828] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.348913] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.355212] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.361840] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.368117] mdt_io01_103 S ffff8b792a37b0c0 0 119188 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.375503] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.378136] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.383331] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.390221] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.396047] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.402539] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.410128] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.415204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.421569] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.428214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.434485] mdt_rdpg02_046 S ffff8b792a37c100 0 119189 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.441857] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.444490] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.449694] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.456592] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.462483] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.468968] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.476719] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.481783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.488059] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.494695] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.501004] mdt_rdpg01_045 S ffff8b792a37d140 0 119190 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.508374] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.511010] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.517621] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.524531] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.530339] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.536841] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.544432] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.549497] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.555774] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.562401] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.568672] mdt_rdpg02_047 S ffff8b792a37e180 0 119191 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.576026] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.578660] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.583856] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.590745] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.596540] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.603025] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.610596] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.615656] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.621934] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.628553] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.634826] mdt_rdpg00_041 S ffff8b7ccda28000 0 119192 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.642190] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.644823] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.650013] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.656894] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.662690] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.669176] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.676748] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.681810] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.688087] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.694707] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.700980] mdt_rdpg01_046 S ffff8b7ccda29040 0 119193 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.708335] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.710966] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.716156] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.723038] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.728837] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.735323] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.742917] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.747974] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.754252] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.760874] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.767156] mdt_io01_104 S ffff8b7ccda2a080 0 119194 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.774517] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.777150] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.782346] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.789233] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.795040] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.801525] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.809121] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:16 2019][2937718.814180] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.820456] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:16 2019][2937718.827086] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:16 2019][2937718.833530] mdt_rdpg02_048 S ffff8b7ccda2b0c0 0 119195 2 0x00000080 [Fri Apr 12 11:13:16 2019][2937718.841035] Call Trace: [Fri Apr 12 11:13:16 2019][2937718.843670] [] schedule+0x29/0x70 [Fri Apr 12 11:13:16 2019][2937718.848862] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:16 2019][2937718.855749] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:16 2019][2937718.861553] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937718.868062] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937718.875658] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937718.880720] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937718.887002] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937718.893625] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937718.899903] mdt_rdpg00_042 S ffff8b7ccda2c100 0 119196 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937718.907276] Call Trace: [Fri Apr 12 11:13:17 2019][2937718.909909] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937718.915119] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937718.922177] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937718.927999] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937718.934499] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937718.942085] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937718.947159] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937718.953445] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937718.960062] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937718.966333] mdt_rdpg01_047 S ffff8b7ccda2d140 0 119197 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937718.973698] Call Trace: [Fri Apr 12 11:13:17 2019][2937718.976328] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937718.981525] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937718.988402] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937718.994199] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.000714] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.008307] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.013371] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.019646] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.026271] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.032539] mdt_rdpg00_043 S ffff8b7ccda2e180 0 119198 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.039894] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.042530] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.047727] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.054610] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.060416] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.066939] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.074528] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.079586] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.085866] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.092485] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.098774] mdt_rdpg02_049 S ffff8b7a85ad0000 0 119199 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.106148] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.108816] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.114012] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.120904] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.126696] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.133178] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.140748] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.145810] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.152084] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.158705] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.164979] mdt_io01_105 S ffff8b7a85ad1040 0 119200 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.172343] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.174975] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.180166] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.187048] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.192841] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.199321] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.206893] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.211953] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.218229] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.224850] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.231124] mdt_rdpg01_048 S ffff8b7a85ad2080 0 119201 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.238486] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.241120] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.246309] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.253194] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.258990] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.265482] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.273058] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.278117] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.284392] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.291013] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.297286] mdt_rdpg00_044 S ffff8b5b3103b0c0 0 119202 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.304641] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.307272] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.312465] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.319351] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.325167] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.331658] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.339249] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.344310] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.350612] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.357242] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.363515] mdt_rdpg01_049 S ffff8b5b3103c100 0 119203 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.370883] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.373528] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.378718] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.385600] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.391417] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.397911] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.405515] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.410585] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.416898] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.423547] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.429822] mdt_io01_106 S ffff8b5b3103d140 0 119204 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.437186] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.439818] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.445013] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.451915] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.457724] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.464247] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.471853] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.476919] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.483195] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.489819] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.496108] mdt_rdpg01_050 S ffff8b5b3103e180 0 119205 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.503487] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.506129] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.511330] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.518308] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.524143] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.530633] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.538218] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.543280] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.549560] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.556199] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.562503] mdt_rdpg00_045 S ffff8b598d0d8000 0 119206 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.569882] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.572517] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.577732] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.584748] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.590608] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.597163] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.605041] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.610244] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.616520] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.623164] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.629443] mdt_io01_107 S ffff8b5736bca080 0 119207 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.636828] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.639468] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.644708] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.651757] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.657574] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.664058] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.671665] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.676736] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.683357] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.690051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.696320] mdt_rdpg03_037 S ffff8b5736bcb0c0 0 119208 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.703752] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.706452] [] ? tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.713692] [] ? schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.719056] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.727118] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.734172] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.739979] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.746634] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.754237] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.759496] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.765779] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.772675] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.778947] mdt_rdpg00_046 S ffff8b598d0d9040 0 119209 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.786311] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.788946] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.794154] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.801035] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:17 2019][2937719.806828] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.813606] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.821242] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:17 2019][2937719.826306] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.832579] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:17 2019][2937719.839200] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:17 2019][2937719.845476] mdt_rdpg01_051 S ffff8b598d0da080 0 119210 2 0x00000080 [Fri Apr 12 11:13:17 2019][2937719.852838] Call Trace: [Fri Apr 12 11:13:17 2019][2937719.855472] [] schedule+0x29/0x70 [Fri Apr 12 11:13:17 2019][2937719.860664] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:17 2019][2937719.867548] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937719.873339] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937719.879836] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937719.887420] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937719.892478] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937719.898772] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937719.905404] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937719.911672] mdt_rdpg02_050 S ffff8b598d0db0c0 0 119211 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937719.919047] Call Trace: [Fri Apr 12 11:13:18 2019][2937719.921696] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937719.926897] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937719.933782] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937719.939614] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937719.946126] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937719.953724] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937719.958807] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937719.965119] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937719.971756] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937719.978036] mdt_rdpg03_038 S ffff8b598d0dc100 0 119212 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937719.985409] Call Trace: [Fri Apr 12 11:13:18 2019][2937719.988044] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937719.993247] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.001318] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.008377] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.014207] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.020907] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.028489] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.033732] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.040006] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.046802] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.053096] mdt_rdpg01_052 S ffff8b598d0dd140 0 119213 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.060508] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.063161] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.068368] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.075258] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.081097] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.087585] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.095162] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.100223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.106497] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.113123] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.119397] mdt_rdpg00_047 S ffff8b598d0de180 0 119214 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.126792] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.129433] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.134645] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.141542] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.147350] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.153940] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.161536] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.166594] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.172874] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.179509] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.185780] mdt_io01_108 S ffff8b58c84a0000 0 119215 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.193152] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.195784] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.200982] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.207888] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.213691] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.220208] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.227786] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.232844] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.239118] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.245742] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.252013] mdt_rdpg01_053 S ffff8b58c84a1040 0 119216 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.259395] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.262031] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.267248] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.274138] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.279951] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.286445] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.294063] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.299132] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.305412] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.312035] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.318308] mdt_rdpg03_039 S ffff8b58c84a2080 0 119217 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.325669] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.328302] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.333497] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.340410] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.346225] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.352735] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.360310] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.365370] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:13:18 2019][2937720.371642] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.377918] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.384538] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.390810] mdt_io01_109 S ffff8b58c84a30c0 0 119218 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.398166] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.400798] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.405989] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.412894] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.418695] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.425179] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.432755] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.438078] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.444477] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.451115] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.457394] mdt_rdpg00_048 S ffff8b58c84a4100 0 119219 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.464772] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.467406] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.472621] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.479504] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.485308] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.491849] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.499439] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.504517] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.510793] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.517428] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.523725] mdt_rdpg02_051 S ffff8b58c84a5140 0 119220 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.531091] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.533726] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.538928] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.545844] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.551661] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.558171] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.565746] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.570806] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.577082] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.583723] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.589992] mdt_rdpg01_054 S ffff8b58c84a6180 0 119221 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.597357] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.599992] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.605184] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.612064] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.617856] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.624338] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.631909] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.636968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.643264] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.649881] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.656155] mdt_rdpg02_052 S ffff8b54d3978000 0 119222 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.663511] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.666144] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.671332] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.678215] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.684013] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.690536] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.698119] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.703177] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.709454] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.716177] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.722477] mdt_io01_110 S ffff8b54d3979040 0 119223 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.729849] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.732508] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.737704] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.744590] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.750384] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.756875] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.764452] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.769520] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.775810] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.782456] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.788743] mdt_rdpg03_040 S ffff8b54d397a080 0 119224 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.796106] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.798742] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.803946] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.810829] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:18 2019][2937720.816623] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.823130] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:18 2019][2937720.830701] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:18 2019][2937720.835760] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.842036] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:18 2019][2937720.848657] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:18 2019][2937720.854930] mdt_rdpg01_055 S ffff8b54d397b0c0 0 119225 2 0x00000080 [Fri Apr 12 11:13:18 2019][2937720.862286] Call Trace: [Fri Apr 12 11:13:18 2019][2937720.864917] [] schedule+0x29/0x70 [Fri Apr 12 11:13:18 2019][2937720.870125] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937720.877021] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937720.882817] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937720.889320] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937720.896900] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937720.901959] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937720.908245] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937720.914875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937720.921157] mdt_rdpg02_053 S ffff8b7b8da0a400 0 119226 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937720.928527] Call Trace: [Fri Apr 12 11:13:19 2019][2937720.931161] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937720.937661] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937720.944553] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937720.950356] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937720.956863] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937720.964451] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937720.969532] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937720.975822] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937720.982464] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937720.988744] mdt_rdpg00_049 S ffff8b5736bcc100 0 119227 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937720.996103] Call Trace: [Fri Apr 12 11:13:19 2019][2937720.998755] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.003948] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.010837] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.016644] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.023141] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.030720] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.035797] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.042091] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.048728] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.055174] mdt_rdpg00_050 S ffff8b5736bcd140 0 119228 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.062587] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.065220] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.070449] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.077375] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.083181] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.089684] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.097262] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.102323] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.108605] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.115229] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.121497] mdt_rdpg01_056 S ffff8b5736bce180 0 119229 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.128852] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.131495] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.136744] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.143647] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.149443] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.155928] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.163499] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.168572] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.174847] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.181485] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.187759] mdt_rdpg02_054 S ffff8b7cf7ed9040 0 119230 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.195512] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.198154] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.203364] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.210291] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.216164] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.222706] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.230303] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.235367] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.241652] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.248275] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.254565] mdt_rdpg03_041 S ffff8b7cf7eda080 0 119231 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.261924] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.264558] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.269754] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.276668] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.282489] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.288973] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.296544] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.301604] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.307907] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.314558] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.320830] mdt_io01_111 S ffff8b7cf7edb0c0 0 119232 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.328200] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.330835] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.336037] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.342923] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.348718] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.355205] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.362793] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.367856] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.374131] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.380783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.387070] mdt_rdpg00_051 S ffff8b7cf7edc100 0 119233 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.394458] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.397090] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.402287] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.409202] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.415056] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.421558] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.429154] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.434222] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.440494] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.447116] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.453401] mdt_rdpg02_055 S ffff8b7cf7edd140 0 119234 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.460777] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.463412] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.468610] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.475503] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.481307] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.487823] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.495402] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.500467] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.506754] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.513689] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.520090] mdt_rdpg00_052 S ffff8b7cf7ede180 0 119235 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.527450] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.530085] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.535279] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.542186] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.548001] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.554529] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.562135] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.567205] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.573481] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.580107] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.586391] mdt_rdpg01_057 S ffff8b7601b40000 0 119236 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.593762] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.596397] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.601652] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.608538] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.614343] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.620838] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.628436] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.633504] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.639780] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.646399] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.652675] mdt_rdpg00_053 S ffff8b7601b41040 0 119237 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.660045] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.662681] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.667874] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.674762] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.680553] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.687042] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.694615] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.699678] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.705962] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.712600] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.718872] mdt_rdpg03_042 S ffff8b7601b42080 0 119238 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.726260] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.728924] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.734167] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.741059] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.746882] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.753368] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.760940] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.766002] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.772289] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.778920] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.785188] mdt_rdpg02_056 S ffff8b7601b430c0 0 119239 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.792552] Call Trace: [Fri Apr 12 11:13:19 2019][2937721.795185] [] schedule+0x29/0x70 [Fri Apr 12 11:13:19 2019][2937721.800403] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.807298] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:19 2019][2937721.813096] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.819595] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:19 2019][2937721.827189] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:19 2019][2937721.832262] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.838539] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:19 2019][2937721.845193] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:19 2019][2937721.851492] mdt_rdpg03_043 S ffff8b6cf75ea080 0 119240 2 0x00000080 [Fri Apr 12 11:13:19 2019][2937721.858883] Call Trace: [Fri Apr 12 11:13:20 2019][2937721.861522] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937721.866720] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937721.873607] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937721.879415] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937721.885903] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937721.893479] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937721.898547] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937721.904822] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937721.911441] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937721.917725] mdt_rdpg00_054 D ffff8b6cf75ed140 0 119241 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937721.925792] Call Trace: [Fri Apr 12 11:13:20 2019][2937721.928469] [] schedule_preempt_disabled+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937721.935096] [] __mutex_lock_slowpath+0xc7/0x1d0 [Fri Apr 12 11:13:20 2019][2937721.941468] [] mutex_lock+0x1f/0x2f [Fri Apr 12 11:13:20 2019][2937721.946849] [] ? tgt_request_handle+0xca8/0x1580 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937721.954119] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937721.961873] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [Fri Apr 12 11:13:20 2019][2937721.970632] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937721.978743] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937721.985749] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:20 2019][2937721.992205] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:20 2019][2937721.998268] [] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.004926] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.012531] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.017787] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.024062] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.030865] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.037137] mdt_rdpg02_057 S ffff8b6cf75ee180 0 119242 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.044516] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.047181] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.052380] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.059264] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.065060] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.071558] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.079150] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.084224] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.090511] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.097157] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.103440] mdt_rdpg01_058 S ffff8b6cf75ec100 0 119243 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.111161] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.113913] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.119125] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.126041] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.131839] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.138332] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.145909] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.150968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.157241] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.163863] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.170136] mdt_rdpg03_044 S ffff8b6cf75e9040 0 119244 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.177500] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.180132] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.185324] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.192206] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.198000] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.204489] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.212067] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.217129] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.223408] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.230032] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.236319] mdt_rdpg01_059 S ffff8b6cf75e8000 0 119245 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.243689] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.246336] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.251558] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.258440] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.264244] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.270729] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.278524] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.283589] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:13:20 2019][2937722.289863] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.296140] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.302757] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.309030] mdt_rdpg02_058 S ffff8b7601b44100 0 119246 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.316387] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.319039] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.324249] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.331164] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.337003] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.343493] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.351070] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.356148] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.362434] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.369050] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.375323] mdt_rdpg00_055 S ffff8b7601b45140 0 119247 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.382679] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.385310] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.390506] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.397390] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.403187] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.409674] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.417248] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.422311] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.428597] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.435225] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.441497] mdt_rdpg03_045 S ffff8b7601b46180 0 119248 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.448863] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.451510] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.456702] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.463593] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.469414] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.475900] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.483474] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.488534] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.494823] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.501450] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.507721] mdt_rdpg02_059 S ffff8b70b1e60600 0 119249 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.515083] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.517717] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.522938] [] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.530989] [] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.538049] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.543841] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.550536] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.558109] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.563344] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.569621] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.576412] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.582687] mdt_rdpg00_056 S ffff8b55c54d5140 0 119250 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.590069] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.592701] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.597893] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.604772] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.610566] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.617049] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.624621] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.629681] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.635956] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.642575] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.648856] mdt_rdpg03_046 S ffff8b55c54d0000 0 119251 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.656214] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.658845] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.664034] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.670910] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.676703] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.683182] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.690754] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.695815] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.702083] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.708704] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.714978] mdt_rdpg00_057 S ffff8b55c54d4100 0 119252 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.722342] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.724973] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.730162] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.737047] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.742838] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.749321] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.756891] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.761954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.768231] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.774850] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.781123] mdt_rdpg03_047 S ffff8b55c54d2080 0 119253 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.788478] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.791110] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.796303] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.803199] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:20 2019][2937722.809003] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.815497] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:20 2019][2937722.823074] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:20 2019][2937722.828153] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.834428] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:20 2019][2937722.841049] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:20 2019][2937722.847321] mdt_rdpg00_058 S ffff8b55c54d1040 0 119254 2 0x00000080 [Fri Apr 12 11:13:20 2019][2937722.854695] Call Trace: [Fri Apr 12 11:13:20 2019][2937722.857325] [] schedule+0x29/0x70 [Fri Apr 12 11:13:20 2019][2937722.862515] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937722.869400] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937722.875217] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937722.881704] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937722.889283] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937722.894341] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937722.900617] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937722.907237] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937722.913514] mdt_rdpg03_048 S ffff8b55c54d30c0 0 119255 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937722.920873] Call Trace: [Fri Apr 12 11:13:21 2019][2937722.923507] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937722.928694] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937722.935572] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937722.941365] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937722.947843] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937722.955422] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937722.960479] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937722.966756] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937722.973377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937722.979659] mdt_rdpg00_059 S ffff8b55c54d6180 0 119256 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937722.987023] Call Trace: [Fri Apr 12 11:13:21 2019][2937722.989664] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937722.994855] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.001751] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.007588] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.014080] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.021660] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.026738] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.033015] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.039643] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.045916] mdt_rdpg03_049 S ffff8b51b228b0c0 0 119257 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.053294] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.055928] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.061133] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.068085] [] ? ptlrpc_server_post_idle_rqbds+0x7e/0xf0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.076029] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.081818] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.088300] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.095893] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.100952] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.107238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.113861] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.120147] mdt_rdpg03_050 S ffff8b681f71e180 0 119258 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.127521] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.130160] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.135352] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.142233] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.148024] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.154529] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.162132] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.167194] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.173480] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.180110] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.186381] mdt_rdpg03_051 S ffff8b681f71b0c0 0 119259 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.193752] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.196398] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.201597] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.208485] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.214285] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.220775] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.228354] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.233419] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.239701] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.246345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.252641] mdt_rdpg03_052 S ffff8b681f71c100 0 119260 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.260000] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.262634] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.267830] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.274736] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.280791] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.287277] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.294866] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.299937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.306238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.312868] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.319140] mdt_rdpg03_053 S ffff8b681f71a080 0 119261 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.326511] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.329154] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.334351] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.341233] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.347024] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.353507] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.361090] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.366160] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.372435] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.379056] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.385359] mdt_rdpg03_054 S ffff8b681f71d140 0 119262 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.392775] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.395421] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.400627] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.407986] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.414040] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.420576] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.428171] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.433231] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.439506] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.446125] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.452400] mdt_rdpg03_055 S ffff8b889f7eb0c0 0 119263 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.459780] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.462416] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.467628] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.474534] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.480342] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.486833] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.494410] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.499472] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.505758] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.512387] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.518660] mdt_rdpg03_056 S ffff8b889f7ee180 0 119264 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.526020] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.528656] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.533845] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.540726] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.546522] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.553006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.560581] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.565642] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.571917] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.578536] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.584811] mdt_rdpg03_057 S ffff8b889f7e9040 0 119265 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.592176] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.594807] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.599994] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.606872] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.612730] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.619229] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.626808] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.631868] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.638143] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.644763] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.651034] mdt_rdpg03_058 S ffff8b889f7ec100 0 119266 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.658399] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.661031] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.666222] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.673122] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.678914] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.685394] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.692968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.698027] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.704303] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.710929] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.717207] mdt_rdpg03_059 S ffff8b889f7ed140 0 119267 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.724569] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.727203] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.732397] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.739274] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.745070] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.751560] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.759139] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.764204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.770485] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.777108] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.783379] mdt_io02_111 S ffff8b5da98230c0 0 119269 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.790779] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.793419] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.798608] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.805532] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:21 2019][2937723.811346] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.817835] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:21 2019][2937723.825419] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:21 2019][2937723.830476] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.836750] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:21 2019][2937723.843376] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:21 2019][2937723.849647] mdt_fld_0002 S ffff8b8b083bd140 0 119277 2 0x00000080 [Fri Apr 12 11:13:21 2019][2937723.857026] Call Trace: [Fri Apr 12 11:13:21 2019][2937723.859680] [] schedule+0x29/0x70 [Fri Apr 12 11:13:21 2019][2937723.864868] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937723.871749] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937723.877545] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937723.884028] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937723.891604] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937723.896665] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937723.902941] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937723.909583] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937723.915862] mdt_fld_0003 S ffff8b8b083ba080 0 119284 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937723.923223] Call Trace: [Fri Apr 12 11:13:22 2019][2937723.925854] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937723.931049] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937723.937929] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937723.943732] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937723.950232] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937723.957820] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937723.962884] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937723.969158] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937723.975778] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937723.982059] ldlm_bl_03 S ffff8b8b083be180 0 119285 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937723.989439] Call Trace: [Fri Apr 12 11:13:22 2019][2937723.992072] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:22 2019][2937723.997973] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:22 2019][2937724.005385] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.010577] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.017649] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.023442] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.031047] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.036107] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.042392] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.049021] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.055294] mdt_out03_002 S ffff8b8b083bc100 0 119288 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.062675] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.065308] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.070507] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.077391] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.083379] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.089881] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.097464] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.102542] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.108834] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.115453] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.121727] ldlm_cn03_006 S ffff8b8b083b9040 0 119290 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.129089] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.131721] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.136915] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.143798] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.149616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.156104] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.163685] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.168741] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.175020] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.181640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.187913] ldlm_cn00_006 S ffff8b8b083bb0c0 0 119315 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.195276] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.197909] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.203100] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.209984] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.215817] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.222300] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.229885] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.234953] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.241238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.247882] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.254158] ldlm_cn01_008 S ffff8b6ce6b81040 0 119542 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.261545] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.264185] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.269382] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.276280] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.282090] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.288584] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.296167] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.301223] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.307502] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.314121] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.320392] ldlm_cn00_007 S ffff8b6ce6b86180 0 119543 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.327750] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.330379] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.335573] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.342455] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.348260] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.354756] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.362336] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.367396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.373672] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.380290] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.386566] ldlm_bl_04 S ffff8b6ce6b85140 0 119549 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.393929] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.396591] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:22 2019][2937724.404011] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.409207] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.416449] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.422255] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.429832] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.434894] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.441173] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.447801] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.454082] ldlm_cn03_007 S ffff8b8cceb95140 0 119782 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.461481] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.464123] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.469326] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.476213] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.482010] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.488535] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.496110] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.501170] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.507448] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.514068] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.520341] ldlm_cn03_008 S ffff8b8b70a4c100 0 119787 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.527703] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.530337] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.535530] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.542435] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.548231] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.554716] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.562300] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.567360] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.573645] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.580266] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.586540] ldlm_cn02_007 S ffff8b5da99330c0 0 119867 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.593918] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.596572] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.601773] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.608676] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.614473] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.620960] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.628544] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.633608] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.639885] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.646507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.652780] ldlm_cn02_008 S ffff8b5da9931040 0 119914 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.660151] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.662789] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.667994] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.674876] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.680686] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.687177] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.694757] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.699816] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.706089] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.712709] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.718984] ldlm_cn00_008 S ffff8b8b70a48000 0 119915 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.726358] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.729000] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.734204] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.741089] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.746882] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.753370] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.760954] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.766014] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.772287] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.778908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.785193] mdt_out00_002 S ffff8b8b70a4a080 0 119947 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.792589] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.795222] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.800456] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.807356] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:22 2019][2937724.813161] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.819645] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.827220] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:22 2019][2937724.832288] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.838573] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:22 2019][2937724.845203] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:22 2019][2937724.851475] ldlm_cn00_009 S ffff8b745065a080 0 120497 2 0x00000080 [Fri Apr 12 11:13:22 2019][2937724.858857] Call Trace: [Fri Apr 12 11:13:22 2019][2937724.861514] [] schedule+0x29/0x70 [Fri Apr 12 11:13:22 2019][2937724.866709] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:22 2019][2937724.873596] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937724.879390] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937724.885877] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937724.893469] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937724.898530] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937724.904807] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937724.911435] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937724.917706] ldlm_cn03_009 S ffff8b745065e180 0 120503 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937724.925077] Call Trace: [Fri Apr 12 11:13:23 2019][2937724.927711] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937724.932944] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937724.939858] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937724.945653] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937724.952137] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937724.959721] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937724.964781] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937724.971055] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937724.977678] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937724.983948] ldlm_cn01_009 S ffff8b5c2a71d140 0 120936 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937724.991338] Call Trace: [Fri Apr 12 11:13:23 2019][2937724.993975] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937724.999176] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.006066] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.011876] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.018394] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.025968] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.031054] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.037370] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.044059] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.050355] ldlm_cn02_009 S ffff8b6cf2aba080 0 121118 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.057726] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.060412] [] ? keys_fini+0xb1/0x1d0 [obdclass] [Fri Apr 12 11:13:23 2019][2937725.066861] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.072084] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.078968] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.084771] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.091280] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.098854] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.103914] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.110191] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.116824] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.123142] ldlm_cn01_010 S ffff8b5c2a71b0c0 0 121244 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.130500] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.133136] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.138327] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.145245] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.151054] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.157566] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.165139] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.170244] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.176521] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.183142] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.189439] ldlm_cn02_010 S ffff8b5c2a718000 0 121492 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.196838] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.199471] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.204681] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.211570] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.217381] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.223990] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.231576] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.236662] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.242935] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.249568] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.255845] ldlm_cn00_010 S ffff8b79fc4a8000 0 121722 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.263216] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.265849] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.271047] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.277942] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.283744] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.290229] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.297814] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.302896] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.309176] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.315820] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.322097] ldlm_cn00_011 S ffff8b79fc4ab0c0 0 121730 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.329470] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.332116] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.337327] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.344224] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.350018] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.356514] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.364089] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.369155] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.375432] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.382055] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.388328] kworker/33:0 S ffff8b65a1298000 0 121822 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.395715] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.398351] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:23 2019][2937725.404546] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.409726] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:23 2019][2937725.415502] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:23 2019][2937725.422228] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.427282] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.433562] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.440177] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.446446] ldlm_cb03_002 S ffff8b633cf2d140 0 122130 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.453819] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.456461] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.461666] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.468553] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.474386] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.480924] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.488654] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.493761] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.500038] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.506678] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.512966] ldlm_cb01_002 S ffff8b633cf2e180 0 122132 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.520339] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.522987] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.528192] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.535080] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.540872] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.547357] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.554936] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.560010] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.566287] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.572914] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.579188] ldlm_cb02_002 S ffff8b5c2a719040 0 122226 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.586568] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.589231] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.594418] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.601321] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.607115] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.613616] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.621213] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.626281] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.632550] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.639170] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.645446] ldlm_cn01_011 R running task 0 122529 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.652808] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.655439] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.660634] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.667513] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.673306] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.679787] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.687360] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.692421] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.698695] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.705317] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.711591] ldlm_cb03_003 S ffff8b5cf70ba080 0 122541 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.718953] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.721584] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.726777] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.733659] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.739450] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.745956] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.753545] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.758611] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.764893] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.771517] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.777790] ldlm_cn01_012 S ffff8b633cf2b0c0 0 122661 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.785162] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.787813] [] schedule+0x29/0x70 [Fri Apr 12 11:13:23 2019][2937725.793005] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.799886] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:23 2019][2937725.805694] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.812184] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:23 2019][2937725.819757] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:23 2019][2937725.824826] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.831103] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:23 2019][2937725.837730] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:23 2019][2937725.844003] ldlm_cb00_002 S ffff8b633cf2a080 0 122745 2 0x00000080 [Fri Apr 12 11:13:23 2019][2937725.851376] Call Trace: [Fri Apr 12 11:13:23 2019][2937725.854532] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937725.859870] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937725.866778] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937725.872581] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937725.879076] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937725.886661] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937725.891727] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937725.898022] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937725.904660] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937725.910590] LustreError: dumping log to /tmp/lustre-log.1555092804.118310 [Fri Apr 12 11:13:24 2019][2937725.917887] ldlm_cn03_010 S ffff8b6a043ca080 0 122828 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937725.925243] Call Trace: [Fri Apr 12 11:13:24 2019][2937725.927879] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937725.933072] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937725.939949] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937725.945743] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937725.952242] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937725.959823] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937725.964893] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937725.971171] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937725.977800] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937725.984071] kworker/4:1 S ffff8b5cf70be180 0 122919 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937725.991457] Call Trace: [Fri Apr 12 11:13:24 2019][2937725.994100] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937725.999249] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:24 2019][2937726.005012] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:24 2019][2937726.011717] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.016789] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.023084] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.029705] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.035976] ldlm_cn01_013 S ffff8b8cf4732080 0 123222 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.043357] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.046007] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.051201] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.058096] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.063908] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.070394] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.077975] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.083057] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.089367] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.095997] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.102269] ldlm_cn01_014 S ffff8b8cf4735140 0 123223 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.109641] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.112281] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.117475] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.124354] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.130142] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.136626] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.144211] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.149271] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.155545] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.162168] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.168441] ldlm_cn01_015 S ffff8b8cf4730000 0 123224 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.175801] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.178437] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.183633] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.190519] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:24 2019][2937726.196981] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.202854] [] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.209533] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.217108] [] ? kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.222342] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.228617] [] ? ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.235408] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.241682] ldlm_cn01_016 S ffff8b8cf4731040 0 123225 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.249047] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.251688] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.256889] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.263771] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.269567] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.276083] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.283663] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.288732] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.295023] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.301644] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.307915] kworker/37:2 S ffff8b6315fb0000 0 123397 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.315284] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.317921] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:24 2019][2937726.324116] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.329261] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:24 2019][2937726.335016] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:24 2019][2937726.341724] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.346784] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.353055] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.359673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.365947] mdt_seqm_0002 S ffff8b60c02fb0c0 0 123405 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.373303] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.375934] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.381126] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.388007] [] ? _raw_spin_unlock_irqrestore+0x15/0x20 [Fri Apr 12 11:13:24 2019][2937726.394988] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.400782] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.407268] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.414842] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.419904] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.426180] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.432800] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.439069] ldlm_cn00_012 S ffff8b60c02fe180 0 123637 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.446441] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.449073] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.454268] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.461158] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.466951] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.473452] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.481031] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.486091] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.492366] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.498989] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.505259] ldlm_cn02_011 S ffff8b60c02fc100 0 123638 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.512630] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.515264] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.520456] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.527336] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.533126] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.539611] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.547192] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.552252] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.558526] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.565146] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.571422] ldlm_cn00_013 S ffff8b888ff52080 0 123639 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.578786] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.581415] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.586607] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.593489] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.599281] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.605761] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.613336] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.618396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.624691] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.631310] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.637585] ldlm_cn03_011 S ffff8b888ff56180 0 123640 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.644948] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.647576] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.652759] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.659635] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.665427] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.671910] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.679485] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.684543] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.690816] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.697436] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.703703] ldlm_cn03_012 S ffff8b8697f8a080 0 123641 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.711058] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.713690] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.718877] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.725755] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.731545] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.738024] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.745600] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.750662] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.756937] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.763557] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.769822] ldlm_cn03_013 S ffff8b8697f8b0c0 0 123642 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.777178] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.779809] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.784998] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.791881] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:24 2019][2937726.797676] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.804170] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.811748] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:24 2019][2937726.816807] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.823081] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:24 2019][2937726.829703] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:24 2019][2937726.835976] ldlm_cn02_012 S ffff8b5cf70bd140 0 123804 2 0x00000080 [Fri Apr 12 11:13:24 2019][2937726.843333] Call Trace: [Fri Apr 12 11:13:24 2019][2937726.845963] [] schedule+0x29/0x70 [Fri Apr 12 11:13:24 2019][2937726.851152] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:24 2019][2937726.858027] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937726.863819] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937726.870300] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937726.877876] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937726.882935] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937726.889212] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937726.895832] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937726.902104] ldlm_cn01_017 S ffff8b5cf70bb0c0 0 123805 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937726.909460] Call Trace: [Fri Apr 12 11:13:25 2019][2937726.912092] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937726.917280] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937726.924170] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937726.929956] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937726.934676] LustreError: dumping log to /tmp/lustre-log.1555092805.117558 [Fri Apr 12 11:13:25 2019][2937726.943402] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937726.950972] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937726.956030] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937726.962298] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937726.968921] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937726.975193] ldlm_cn01_018 S ffff8b5cf70b9040 0 123806 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937726.982548] Call Trace: [Fri Apr 12 11:13:25 2019][2937726.985180] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937726.990368] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937726.997246] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.003040] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.009526] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.017100] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.022158] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.028425] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.035047] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.041320] ldlm_cn01_019 S ffff8b8ae8abd140 0 123807 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.048693] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.051323] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.056513] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.063390] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.069180] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.075665] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.083236] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.088295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.094562] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.101185] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.107475] ldlm_cb01_003 S ffff8b62607b8000 0 124105 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.114832] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.117461] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.122650] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.129527] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.135319] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.141796] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.149373] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.154433] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.160709] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.167345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.173612] ldlm_cb00_003 S ffff8b62607be180 0 124118 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.180969] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.183603] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.188787] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.195662] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.201457] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.207938] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.215511] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.220569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.226847] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.233467] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.239741] kworker/46:2 S ffff8b792207a080 0 125091 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.247110] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.249747] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:25 2019][2937727.255931] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.261078] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:25 2019][2937727.266835] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:25 2019][2937727.273542] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.278606] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.284893] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.291508] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.297783] ldlm_bl_05 S ffff8b792207c100 0 125543 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.305147] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.307779] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:25 2019][2937727.313644] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:25 2019][2937727.321045] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.326231] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.333284] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.339075] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.346662] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.351717] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.357992] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.364614] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.370888] ldlm_cn00_014 S ffff8b7a6d3cb0c0 0 126370 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.378251] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.380882] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.386073] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.392955] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.398751] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.405229] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.412802] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.417862] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.424143] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.430762] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.437031] ldlm_cn01_020 S ffff8b7a6d3c8000 0 126371 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.444399] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.447036] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.452243] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.459141] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.464952] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.471440] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.479017] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.484079] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.490355] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.496974] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.503247] ldlm_cn01_021 S ffff8b7a6d3c9040 0 126372 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.510606] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.513234] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.518421] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.525300] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.531092] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.537575] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.545145] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.550204] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.556474] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.563094] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.569366] ldlm_cn02_013 S ffff8b8c6125e180 0 126466 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.576723] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.579353] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.584541] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.591417] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.597211] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.603694] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.611266] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.616324] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.622602] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.629230] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.635505] ldlm_cn01_022 S ffff8b8c6125d140 0 126467 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.642866] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.645500] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.650690] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.657572] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.663366] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.669847] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.677418] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.682478] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.688757] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.695375] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.701648] ldlm_cn01_023 S ffff8b8c6125a080 0 126533 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.709077] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.711744] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.716965] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.723875] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.729704] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.736216] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.743801] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.748862] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.755140] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.761776] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.768058] ldlm_cn01_024 S ffff8b8c6125c100 0 126534 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.775436] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.778069] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.783266] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.790160] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.795951] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.802446] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.810025] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:25 2019][2937727.815110] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.821386] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:25 2019][2937727.828008] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:25 2019][2937727.834281] ldlm_cn01_025 S ffff8b7a6d3ce180 0 126535 2 0x00000080 [Fri Apr 12 11:13:25 2019][2937727.841644] Call Trace: [Fri Apr 12 11:13:25 2019][2937727.844274] [] schedule+0x29/0x70 [Fri Apr 12 11:13:25 2019][2937727.849465] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:25 2019][2937727.856349] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:25 2019][2937727.862139] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937727.868624] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937727.876201] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937727.881262] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937727.887544] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937727.894164] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937727.900433] ldlm_cn01_026 S ffff8b7a6d3cd140 0 126536 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937727.907815] Call Trace: [Fri Apr 12 11:13:26 2019][2937727.910448] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937727.915645] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937727.922528] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937727.928322] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937727.934820] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937727.942394] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937727.947464] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937727.953735] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937727.960362] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937727.966631] ldlm_cn01_027 S ffff8b8cf445e180 0 126537 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937727.974004] Call Trace: [Fri Apr 12 11:13:26 2019][2937727.976634] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937727.981838] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937727.988729] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937727.994528] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.001007] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.008594] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.013660] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.019937] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.026584] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.032857] ldlm_cn00_015 S ffff8b745f616180 0 126538 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.040280] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.042914] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.048123] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.055002] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.060807] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.067321] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.074912] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.080279] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.086603] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.093229] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.099504] mdt_out00_003 S ffff8b8ae8aba080 0 126773 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.106877] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.109521] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.114719] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.121620] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.127441] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.133968] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.141551] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.146611] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.152882] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.159507] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.165777] ldlm_cn02_014 S ffff8b5af4e59040 0 126949 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.173134] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.175765] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.180969] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.187868] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.193666] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.200151] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.207730] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.212791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.219073] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.225706] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.231976] ldlm_cn02_015 S ffff8b5af4e5b0c0 0 126950 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.239343] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.241991] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.247184] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.254072] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.259867] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.266356] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.273945] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.279015] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.285291] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.291920] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.298209] ldlm_cn02_016 S ffff8b6784ac0000 0 126997 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.305570] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.308202] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.313398] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.320276] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.326067] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.332550] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.340121] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.345182] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.351458] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.358079] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.364352] ldlm_cn00_016 S ffff8b69caf18000 0 127039 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.371725] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.374356] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.379543] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.386421] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.392211] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.398694] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.406268] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.411329] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.417603] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.424225] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.430515] ldlm_cn00_017 S ffff8b51b228c100 0 127574 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.437871] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.440504] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.445691] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.452567] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.458363] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.467136] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.474731] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.479800] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.486072] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.492693] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.498966] ldlm_cn00_018 S ffff8b51b228d140 0 127575 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.506321] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.508955] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.514147] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.521026] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.526823] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.533309] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.540881] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.545943] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.552238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.558855] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.565237] ldlm_cn00_019 S ffff8b51b228e180 0 127576 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.572607] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.575242] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.580433] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.587314] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.593116] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.599616] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.607193] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.612255] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.618529] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.625149] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.631423] ldlm_bl_06 S ffff8b7671b26180 0 128939 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.638785] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.641417] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:26 2019][2937728.647285] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:26 2019][2937728.654686] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.659873] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.666923] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.672738] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.680318] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.685374] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.691919] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.698616] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.704896] ldlm_bl_07 S ffff8b7671b22080 0 128940 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.712275] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.714917] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:26 2019][2937728.720790] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:26 2019][2937728.728194] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.733382] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.740453] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.746260] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.753837] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.758906] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.765191] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.771812] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.778086] ldlm_bl_08 S ffff8b7671b20000 0 128941 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.785460] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.788100] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:26 2019][2937728.793976] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:26 2019][2937728.801377] [] schedule+0x29/0x70 [Fri Apr 12 11:13:26 2019][2937728.806563] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.813619] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:26 2019][2937728.819408] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:26 2019][2937728.826984] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:26 2019][2937728.832048] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.838334] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:26 2019][2937728.844979] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:26 2019][2937728.851271] ldlm_bl_09 S ffff8b7671b230c0 0 128942 2 0x00000080 [Fri Apr 12 11:13:26 2019][2937728.858657] Call Trace: [Fri Apr 12 11:13:26 2019][2937728.861292] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:26 2019][2937728.867158] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:27 2019][2937728.874564] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937728.879768] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937728.886832] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937728.892621] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937728.900204] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937728.905273] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937728.911547] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937728.918173] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937728.924451] ldlm_bl_10 S ffff8b7671b21040 0 128943 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937728.931814] Call Trace: [Fri Apr 12 11:13:27 2019][2937728.934449] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:27 2019][2937728.940324] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:27 2019][2937728.947727] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937728.952965] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937728.960024] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937728.965811] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937728.973384] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937728.978457] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937728.984734] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937728.991360] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937728.997660] ldlm_bl_11 S ffff8b7671b25140 0 128944 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.005031] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.007663] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:27 2019][2937729.013530] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:27 2019][2937729.020933] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.026141] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.033209] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.039022] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.046639] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.051709] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.057987] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.064632] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.070909] ldlm_bl_12 S ffff8b8c8dec4100 0 128945 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.078267] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.080898] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:27 2019][2937729.086764] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:27 2019][2937729.094165] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.099353] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.106404] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.112191] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.119767] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.124827] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.131095] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.137716] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.143987] ldlm_bl_13 S ffff8b8c8dec6180 0 128946 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.151343] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.153975] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:27 2019][2937729.159840] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:27 2019][2937729.167238] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.172421] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.179476] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.185264] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.192837] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.197899] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.204174] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.210794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.217090] ldlm_bl_14 S ffff8b8c8dec1040 0 128947 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.224458] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.227093] [] ? call_rcu_sched+0x1d/0x20 [Fri Apr 12 11:13:27 2019][2937729.232959] [] ? obd_stale_export_get+0x31/0x190 [obdclass] [Fri Apr 12 11:13:27 2019][2937729.240360] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.245546] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.252595] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.258387] [] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.265963] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.271032] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.277311] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.283937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.290211] ldlm_cn03_014 S ffff8b86d4305140 0 129485 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.297583] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.300216] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.305407] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.312289] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.318096] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.324577] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.332187] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.337244] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.343521] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.350141] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.356414] ldlm_cn03_015 S ffff8b86d4302080 0 129486 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.363770] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.366399] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.371589] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.378464] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.384254] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.390743] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.398322] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.403415] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.409693] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.416315] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.422597] ldlm_cn03_016 S ffff8b86d43030c0 0 129487 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.429979] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.432618] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.437809] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.444693] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.450507] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.456987] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.464563] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.469627] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.475900] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.482524] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.488794] ldlm_cn02_017 S ffff8b8c8dec30c0 0 129939 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.496158] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.498793] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.504013] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.510899] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.516704] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.523188] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.530763] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.535822] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.542096] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.548722] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.555026] ldlm_cn02_018 S ffff8b8697f88000 0 129940 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.562397] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.565030] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.570233] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.577139] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.582930] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.589408] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.596984] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.602043] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.608320] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.614950] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.621223] ldlm_cn02_019 S ffff8b8cceb930c0 0 129941 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.628587] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.631222] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.636427] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.643343] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.649137] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.655634] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.663208] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.668268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.674541] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.681172] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.687456] ldlm_cn02_020 S ffff8b8cceb90000 0 129942 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.694818] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.697451] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.702639] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.709523] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.715317] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.721799] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.729379] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.734439] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.740714] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.747335] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.753607] ldlm_cn00_020 S ffff8b8cceb94100 0 129943 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.760962] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.763593] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.768784] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.775668] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.781460] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.787945] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.795515] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.800578] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.806854] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:27 2019][2937729.813473] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.819753] ldlm_cn00_021 S ffff8b8cceb96180 0 129944 2 0x00000080 [Fri Apr 12 11:13:27 2019][2937729.827109] Call Trace: [Fri Apr 12 11:13:27 2019][2937729.829740] [] schedule+0x29/0x70 [Fri Apr 12 11:13:27 2019][2937729.834935] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.841815] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:27 2019][2937729.847602] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.854083] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:27 2019][2937729.861662] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:27 2019][2937729.866722] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:27 2019][2937729.872997] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937729.879614] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937729.885881] ldlm_cn00_022 S ffff8b8cceb92080 0 129945 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937729.893239] Call Trace: [Fri Apr 12 11:13:28 2019][2937729.895869] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937729.901058] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937729.907943] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937729.913750] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937729.920240] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937729.927816] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937729.932875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937729.939152] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937729.945789] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937729.952061] ldlm_cn02_021 S ffff8b8cceb91040 0 129946 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937729.959424] Call Trace: [Fri Apr 12 11:13:28 2019][2937729.962059] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937729.967259] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937729.974142] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937729.979940] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937729.986431] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937729.994004] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937729.999074] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.005349] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.011969] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.018242] ldlm_cn00_023 S ffff8b8cf5489040 0 129947 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.025608] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.028239] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.033430] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.040315] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937730.046108] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.052593] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.060167] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.065247] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.071525] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.078144] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.084417] ldlm_cn00_024 S ffff8b58d53dd140 0 129948 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.091805] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.094450] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.099654] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.106565] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937730.112359] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.118860] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.126453] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.131513] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.137807] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.144430] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.150730] ldlm_cn01_028 S ffff8b58d53db0c0 0 129949 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.158099] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.160737] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.165942] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.172832] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937730.178639] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.185169] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.192755] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.197823] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.204099] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.210720] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.217000] ldlm_cn01_029 S ffff8b58d53dc100 0 129951 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.224365] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.227007] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.232196] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.239091] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937730.244891] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.251379] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.258957] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.264018] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.270296] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.276935] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.283216] ldlm_cn01_030 S ffff8b58d53da080 0 129952 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.290589] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.293221] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.298409] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.305294] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937730.311105] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.317603] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.325182] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.330240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.336521] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.343160] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.349439] ldlm_cn01_031 S ffff8b58d53de180 0 129953 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.356828] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.359463] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.364657] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.371552] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937730.377344] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.383833] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.391409] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.396467] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.402775] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.409436] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.415707] kworker/38:2 S ffff8b73f7e46180 0 1680 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.422989] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.425623] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.430782] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:28 2019][2937730.436563] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:28 2019][2937730.443270] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.448332] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.454605] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.461228] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.467498] mdt_out03_003 S ffff8b6a82fc4100 0 1712 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.474793] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.477437] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.482641] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.489528] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937730.495332] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.501820] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.509398] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.514458] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.520738] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.527365] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.533635] mdt_out03_004 S ffff8b6a82fc5140 0 1919 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.540929] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.543562] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.548795] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.555679] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:28 2019][2937730.561475] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.567962] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:28 2019][2937730.575544] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.580613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.586885] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.593519] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.599799] kmmpd-dm-4 S ffff8b6a82fc1040 0 2215 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.607084] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.609726] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.614872] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:13:28 2019][2937730.620888] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:13:28 2019][2937730.627435] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Fri Apr 12 11:13:28 2019][2937730.634413] [] schedule_timeout_interruptible+0x1e/0x20 [Fri Apr 12 11:13:28 2019][2937730.641501] [] kmmpd+0x216/0x430 [ldiskfs] [Fri Apr 12 11:13:28 2019][2937730.647445] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Fri Apr 12 11:13:28 2019][2937730.654164] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.659222] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.665497] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.672117] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.678391] kmmpd-dm-2 S ffff8b6a82fc0000 0 2216 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.685673] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.688309] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.693456] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:13:28 2019][2937730.699471] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:13:28 2019][2937730.706017] [] ? write_mmp_block+0x12a/0x160 [ldiskfs] [Fri Apr 12 11:13:28 2019][2937730.712999] [] schedule_timeout_interruptible+0x1e/0x20 [Fri Apr 12 11:13:28 2019][2937730.720091] [] kmmpd+0x216/0x430 [ldiskfs] [Fri Apr 12 11:13:28 2019][2937730.726222] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [Fri Apr 12 11:13:28 2019][2937730.732937] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.738003] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.744298] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.750917] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.757188] jbd2/dm-2-8 S ffff8b8697f8d140 0 2218 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.764455] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.767090] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.772258] [] kjournald2+0x203/0x260 [jbd2] [Fri Apr 12 11:13:28 2019][2937730.778407] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:13:28 2019][2937730.784439] [] ? commit_timeout+0x10/0x10 [jbd2] [Fri Apr 12 11:13:28 2019][2937730.790888] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.795978] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.802256] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.808882] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.815155] ldiskfs-rsv-con S ffff8b6784ac1040 0 2219 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.822437] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.825074] [] schedule+0x29/0x70 [Fri Apr 12 11:13:28 2019][2937730.830229] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:13:28 2019][2937730.836069] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:13:28 2019][2937730.842260] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:28 2019][2937730.847317] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.853595] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:28 2019][2937730.860222] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:28 2019][2937730.866498] osp_up2-3 S ffff8b6784ac6180 0 2223 2 0x00000080 [Fri Apr 12 11:13:28 2019][2937730.873773] Call Trace: [Fri Apr 12 11:13:28 2019][2937730.876414] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:13:29 2019][2937730.883392] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937730.888547] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:13:29 2019][2937730.895608] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937730.901380] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:13:29 2019][2937730.908611] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937730.913674] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937730.919945] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937730.926567] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937730.932840] osp-pre-2-3 S ffff8b6784ac5140 0 2224 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937730.940116] Call Trace: [Fri Apr 12 11:13:29 2019][2937730.942748] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:29 2019][2937730.948158] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937730.953382] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:29 2019][2937730.960354] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937730.966111] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:29 2019][2937730.972821] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937730.977880] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937730.984153] [] ret_from_fork_nospec_begin+0xe] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937730.997055] osp_up0-3 S ffff8b6784ac2080 0 2226 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.004340] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.006987] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:13:29 2019][2937731.013971] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.019148] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:13:29 2019][2937731.026217] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.031989] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:13:29 2019][2937731.039237] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937731.044299] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.050578] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937731.057200] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.063478] osp-pre-0-3 S ffff8b7da96ca080 0 2227 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.070771] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.073406] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:29 2019][2937731.079369] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.084909] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:29 2019][2937731.091910] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.097689] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:29 2019][2937731.104396] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937731.109457] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.115740] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937731.122364] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.128647] osp_up1-3 S ffff8b89d9ffd140 0 2228 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.135928] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.138567] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:13:29 2019][2937731.145536] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.150691] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:13:29 2019][2937731.157750] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.163506] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:13:29 2019][2937731.170728] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937731.175788] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.182055] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937731.188683] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.194950] osp-pre-1-3 S ffff8b89d9ff8000 0 2229 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.202247] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.204878] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:29 2019][2937731.210283] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.215437] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:29 2019][2937731.222400] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.228159] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:29 2019][2937731.234865] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937731.239925] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.246209] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937731.252857] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.259266] osp-syn-12-3 S ffff8b69caf1b0c0 0 2232 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.266545] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.269177] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:29 2019][2937731.275278] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.280432] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:29 2019][2937731.287743] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.293532] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.300849] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:29 2019][2937731.306518] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:29 2019][2937731.313076] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.320345] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.327626] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.334950] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:29 2019][2937731.341940] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.349714] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.356984] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.364750] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.372420] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:29 2019][2937731.378875] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:29 2019][2937731.384894] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:29 2019][2937731.391453] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.398320] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:29 2019][2937731.404783] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:29 2019][2937731.410460] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:29 2019][2937731.418042] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937731.423101] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.429376] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937731.435997] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.442269] osp-pre-12-3 S ffff8b69caf1d140 0 2233 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.449546] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.452182] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:29 2019][2937731.457613] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.462767] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:29 2019][2937731.469764] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.475571] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:29 2019][2937731.482316] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937731.487384] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.493661] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937731.500280] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.506554] osp-syn-13-3 S ffff8b69caf19040 0 2234 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.513847] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.516481] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:29 2019][2937731.522581] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.527750] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:29 2019][2937731.535071] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.540892] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.548212] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:29 2019][2937731.553883] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:29 2019][2937731.560443] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.567722] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.574979] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.582315] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.590087] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.597333] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.605105] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.612838] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:29 2019][2937731.619286] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:29 2019][2937731.625307] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:29 2019][2937731.631865] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.638688] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:29 2019][2937731.645174] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:29 2019][2937731.650846] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:29 2019][2937731.658426] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937731.663488] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.669780] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937731.676402] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.682672] osp-pre-13-3 S ffff8b69caf1c100 0 2235 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.689950] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.692591] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:29 2019][2937731.697998] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.703171] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:29 2019][2937731.710149] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.715921] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:29 2019][2937731.722629] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:29 2019][2937731.727694] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.733989] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:29 2019][2937731.740613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:29 2019][2937731.746894] osp-syn-39-3 S ffff8b69caf1e180 0 2236 2 0x00000080 [Fri Apr 12 11:13:29 2019][2937731.754179] Call Trace: [Fri Apr 12 11:13:29 2019][2937731.756811] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:29 2019][2937731.762921] [] schedule+0x29/0x70 [Fri Apr 12 11:13:29 2019][2937731.768076] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:29 2019][2937731.775402] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:29 2019][2937731.781180] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:29 2019][2937731.788500] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:29 2019][2937731.794195] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937731.800805] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937731.808051] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:30 2019][2937731.815331] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937731.822665] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937731.830438] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937731.837682] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937731.845447] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937731.853109] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:30 2019][2937731.859565] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:30 2019][2937731.865588] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937731.872154] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:30 2019][2937731.878958] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:30 2019][2937731.885411] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:30 2019][2937731.891083] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:30 2019][2937731.898682] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:30 2019][2937731.903747] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937731.910027] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:30 2019][2937731.916656] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937731.922934] osp-pre-39-3 S ffff8b69caf1a080 0 2237 2 0x00000080 [Fri Apr 12 11:13:30 2019][2937731.930210] Call Trace: [Fri Apr 12 11:13:30 2019][2937731.932878] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:30 2019][2937731.938303] [] schedule+0x29/0x70 [Fri Apr 12 11:13:30 2019][2937731.943455] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:30 2019][2937731.950417] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:30 2019][2937731.956177] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:30 2019][2937731.962879] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:30 2019][2937731.967943] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937731.974217] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:30 2019][2937731.980839] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937731.987110] osp-syn-46-3 S ffff8b8697f8e180 0 2238 2 0x00000080 [Fri Apr 12 11:13:30 2019][2937731.994381] Call Trace: [Fri Apr 12 11:13:30 2019][2937731.997019] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:30 2019][2937732.003814] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:30 2019][2937732.009917] [] schedule+0x29/0x70 [Fri Apr 12 11:13:30 2019][2937732.015067] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:30 2019][2937732.022379] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:30 2019][2937732.028155] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.035470] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937732.042019] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.049267] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.056512] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.063872] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.071636] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.078880] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.086645] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.094309] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:30 2019][2937732.100756] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:30 2019][2937732.106792] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937732.113362] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.120194] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:30 2019][2937732.126646] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:30 2019][2937732.132329] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:30 2019][2937732.139911] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:30 2019][2937732.144973] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.151246] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:30 2019][2937732.157868] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.164144] osp-pre-46-3 S ffff8b8cf445d140 0 2239 2 0x00000080 [Fri Apr 12 11:13:30 2019][2937732.171445] Call Trace: [Fri Apr 12 11:13:30 2019][2937732.174077] [] schedule+0x29/0x70 [Fri Apr 12 11:13:30 2019][2937732.179230] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:30 2019][2937732.186200] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:30 2019][2937732.191970] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:30 2019][2937732.198682] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:30 2019][2937732.203796] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.210071] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:30 2019][2937732.216694] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.222969] osp-syn-10-3 S ffff8b8ae8abb0c0 0 2240 2 0x00000080 [Fri Apr 12 11:13:30 2019][2937732.230267] Call Trace: [Fri Apr 12 11:13:30 2019][2937732.232899] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:30 2019][2937732.239000] [] schedule+0x29/0x70 [Fri Apr 12 11:13:30 2019][2937732.244156] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:30 2019][2937732.251483] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:30 2019][2937732.257262] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.264631] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937732.271204] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.278475] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.285732] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.293086] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.300866] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.308112] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.315897] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.323565] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:30 2019][2937732.330009] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:30 2019][2937732.336042] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937732.342601] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.349418] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:30 2019][2937732.355877] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:30 2019][2937732.361543] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:30 2019][2937732.369114] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:30 2019][2937732.374172] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.380441] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:30 2019][2937732.387066] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.393338] osp-pre-10-3 S ffff8b8ae8ab8000 0 2241 2 0x00000080 [Fri Apr 12 11:13:30 2019][2937732.400615] Call Trace: [Fri Apr 12 11:13:30 2019][2937732.403258] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:30 2019][2937732.408702] [] schedule+0x29/0x70 [Fri Apr 12 11:13:30 2019][2937732.413860] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:30 2019][2937732.420834] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:30 2019][2937732.426589] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:30 2019][2937732.433296] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:30 2019][2937732.438353] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.444634] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:30 2019][2937732.451261] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.457549] osp-syn-11-3 S ffff8b8ae8abe180 0 2242 2 0x00000080 [Fri Apr 12 11:13:30 2019][2937732.464830] Call Trace: [Fri Apr 12 11:13:30 2019][2937732.467494] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:30 2019][2937732.474358] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:30 2019][2937732.480468] [] schedule+0x29/0x70 [Fri Apr 12 11:13:30 2019][2937732.485624] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:30 2019][2937732.492941] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:30 2019][2937732.498722] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.506048] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937732.512622] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.519882] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.527154] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.534500] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.542267] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.549527] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.557291] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.564959] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:30 2019][2937732.571406] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:30 2019][2937732.577429] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937732.583997] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.590815] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:30 2019][2937732.597295] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:30 2019][2937732.603021] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:30 2019][2937732.610591] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:30 2019][2937732.615652] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.621925] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:30 2019][2937732.628549] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.634832] osp-pre-11-3 S ffff8b5ada860000 0 2243 2 0x00000080 [Fri Apr 12 11:13:30 2019][2937732.642108] Call Trace: [Fri Apr 12 11:13:30 2019][2937732.644741] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:30 2019][2937732.650149] [] schedule+0x29/0x70 [Fri Apr 12 11:13:30 2019][2937732.655302] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:30 2019][2937732.662273] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:30 2019][2937732.668029] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:30 2019][2937732.674732] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:30 2019][2937732.679794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.686070] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:30 2019][2937732.692690] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:30 2019][2937732.698964] osp-syn-14-3 S ffff8b57d5eed140 0 2244 2 0x00000080 [Fri Apr 12 11:13:30 2019][2937732.706232] Call Trace: [Fri Apr 12 11:13:30 2019][2937732.708866] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:30 2019][2937732.714966] [] schedule+0x29/0x70 [Fri Apr 12 11:13:30 2019][2937732.720119] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:30 2019][2937732.727433] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:30 2019][2937732.733225] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.740564] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:30 2019][2937732.746274] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937732.752825] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.760082] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.767326] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.774679] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.782467] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.789714] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.797486] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.805148] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:30 2019][2937732.811592] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:30 2019][2937732.817613] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:30 2019][2937732.824171] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:30 2019][2937732.830967] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:30 2019][2937732.837412] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:30 2019][2937732.843096] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:30 2019][2937732.850685] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937732.855752] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937732.862024] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937732.868644] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937732.874912] osp-pre-14-3 S ffff8b57d5eee180 0 2245 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937732.882189] Call Trace: [Fri Apr 12 11:13:31 2019][2937732.884822] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:31 2019][2937732.890226] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937732.895390] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:31 2019][2937732.902403] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:31 2019][2937732.908187] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:31 2019][2937732.914920] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937732.919980] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937732.926260] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937732.932879] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937732.939160] osp-syn-26-3 S ffff8b5b7ffee180 0 2246 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937732.946435] Call Trace: [Fri Apr 12 11:13:31 2019][2937732.949077] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:31 2019][2937732.955882] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:31 2019][2937732.961982] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937732.967173] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:31 2019][2937732.974491] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:31 2019][2937732.980272] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937732.987595] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:31 2019][2937732.993273] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:31 2019][2937732.999821] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.007068] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.014311] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.021644] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.029408] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.036661] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.044437] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.052153] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:31 2019][2937733.058608] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:31 2019][2937733.064642] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:31 2019][2937733.071194] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.078069] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:31 2019][2937733.084617] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:31 2019][2937733.090286] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:31 2019][2937733.097858] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937733.102921] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.109197] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937733.115830] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.122108] osp-pre-26-3 S ffff8b5cf70bc100 0 2247 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937733.129384] Call Trace: [Fri Apr 12 11:13:31 2019][2937733.132016] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:31 2019][2937733.137423] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937733.142580] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:31 2019][2937733.149551] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:31 2019][2937733.155308] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:31 2019][2937733.162022] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937733.167081] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.173358] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937733.179979] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.186251] osp-syn-31-3 S ffff8b792207e180 0 2248 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937733.193528] Call Trace: [Fri Apr 12 11:13:31 2019][2937733.196159] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:31 2019][2937733.202260] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937733.207414] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:31 2019][2937733.214735] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:31 2019][2937733.220547] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.227874] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:31 2019][2937733.233557] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:31 2019][2937733.240202] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.247556] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.254836] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.262206] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.270037] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.277283] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.285060] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.292748] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:31 2019][2937733.299199] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:31 2019][2937733.305219] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:31 2019][2937733.311769] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.318566] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:31 2019][2937733.325009] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:31 2019][2937733.330682] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:31 2019][2937733.338253] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937733.343313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.349587] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937733.356229] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.362501] osp-pre-31-3 S ffff8b792207b0c0 0 2249 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937733.369813] Call Trace: [Fri Apr 12 11:13:31 2019][2937733.372457] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:31 2019][2937733.377864] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937733.383016] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:31 2019][2937733.390002] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:31 2019][2937733.395762] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:31 2019][2937733.402465] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937733.407524] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.413820] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937733.420459] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.426785] osp-syn-42-3 S ffff8b792207d140 0 2250 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937733.434171] Call Trace: [Fri Apr 12 11:13:31 2019][2937733.436847] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:31 2019][2937733.443019] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937733.448213] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:31 2019][2937733.455585] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:31 2019][2937733.461421] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.468783] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:31 2019][2937733.475350] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.482611] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.489854] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.497178] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:31 2019][2937733.504178] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.511956] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.519230] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.527001] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.534682] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:31 2019][2937733.541130] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:31 2019][2937733.547177] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:31 2019][2937733.553758] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.560585] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:31 2019][2937733.567034] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:31 2019][2937733.572707] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:31 2019][2937733.580304] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937733.585372] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.591647] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937733.598267] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.604541] osp-pre-42-3 S ffff8b7922079040 0 2251 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937733.611826] Call Trace: [Fri Apr 12 11:13:31 2019][2937733.614458] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:31 2019][2937733.619866] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937733.625020] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:31 2019][2937733.632001] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:31 2019][2937733.637758] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:31 2019][2937733.644486] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937733.649548] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.655826] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937733.662443] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.668727] osp-syn-9-3 S ffff8b6da99230c0 0 2252 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937733.676031] Call Trace: [Fri Apr 12 11:13:31 2019][2937733.678682] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:31 2019][2937733.684783] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937733.689936] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:31 2019][2937733.697256] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:31 2019][2937733.703030] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.710358] [] ? __cond_resched+0x26/0x30 [Fri Apr 12 11:13:31 2019][2937733.716203] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:31 2019][2937733.722764] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.730036] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.737284] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.744625] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.752391] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.759640] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.767413] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.775075] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:31 2019][2937733.781528] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:31 2019][2937733.787549] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:31 2019][2937733.794108] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:31 2019][2937733.800915] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:31 2019][2937733.807366] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:31 2019][2937733.813037] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:31 2019][2937733.820611] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:31 2019][2937733.825673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.831953] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:31 2019][2937733.838576] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:31 2019][2937733.844845] osp-pre-9-3 S ffff8b6da9920000 0 2253 2 0x00000080 [Fri Apr 12 11:13:31 2019][2937733.852123] Call Trace: [Fri Apr 12 11:13:31 2019][2937733.854760] [] ? osp_statfs_update+0x1ae/0x520 [osp] [Fri Apr 12 11:13:31 2019][2937733.861586] [] schedule+0x29/0x70 [Fri Apr 12 11:13:31 2019][2937733.866755] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:32 2019][2937733.873732] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937733.879493] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:32 2019][2937733.886198] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937733.891261] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937733.897536] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937733.904160] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937733.910440] osp-syn-2-3 S ffff8b6ee1362080 0 2254 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937733.917722] Call Trace: [Fri Apr 12 11:13:32 2019][2937733.920358] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:32 2019][2937733.926466] [] schedule+0x29/0x70 [Fri Apr 12 11:13:32 2019][2937733.931619] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:32 2019][2937733.938939] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937733.944857] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937733.952175] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:32 2019][2937733.957846] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:32 2019][2937733.964406] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:32 2019][2937733.971674] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:32 2019][2937733.978918] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937733.986274] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:32 2019][2937733.994076] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.001324] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.009132] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.016820] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:32 2019][2937734.023266] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:32 2019][2937734.029284] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:32 2019][2937734.035846] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.042660] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:32 2019][2937734.049113] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:32 2019][2937734.054794] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:32 2019][2937734.062373] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937734.067433] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.073705] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937734.080326] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.086603] osp-pre-2-3 S ffff8b6ee1360000 0 2255 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937734.093886] Call Trace: [Fri Apr 12 11:13:32 2019][2937734.096521] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:32 2019][2937734.101933] [] schedule+0x29/0x70 [Fri Apr 12 11:13:32 2019][2937734.107086] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:32 2019][2937734.114100] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937734.119864] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:32 2019][2937734.126780] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937734.131852] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.138127] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937734.144751] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.151022] osp-syn-43-3 S ffff8b83e6350000 0 2256 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937734.158334] Call Trace: [Fri Apr 12 11:13:32 2019][2937734.160966] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:32 2019][2937734.167075] [] schedule+0x29/0x70 [Fri Apr 12 11:13:32 2019][2937734.172242] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:32 2019][2937734.179564] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937734.185343] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.192667] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:32 2019][2937734.199243] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.206506] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.213773] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.221106] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.228867] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.236126] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.243890] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.251554] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:32 2019][2937734.258008] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:32 2019][2937734.264029] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:32 2019][2937734.270598] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.277441] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:32 2019][2937734.283906] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:32 2019][2937734.289578] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:32 2019][2937734.297172] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937734.302245] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.308538] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937734.315176] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.321446] osp-pre-43-3 S ffff8b83e6351040 0 2257 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937734.328725] Call Trace: [Fri Apr 12 11:13:32 2019][2937734.331368] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:32 2019][2937734.336798] [] schedule+0x29/0x70 [Fri Apr 12 11:13:32 2019][2937734.341960] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:32 2019][2937734.348934] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937734.354691] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:32 2019][2937734.361428] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937734.366493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.372767] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937734.379389] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.385677] osp-syn-19-3 S ffff8b83e6352080 0 2258 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937734.392952] Call Trace: [Fri Apr 12 11:13:32 2019][2937734.395583] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:32 2019][2937734.401688] [] schedule+0x29/0x70 [Fri Apr 12 11:13:32 2019][2937734.406861] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:32 2019][2937734.414177] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937734.419968] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.427307] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:32 2019][2937734.433896] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.441168] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.448420] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.455755] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.463522] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.470770] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.478574] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.486262] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:32 2019][2937734.492717] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:32 2019][2937734.498738] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:32 2019][2937734.505296] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.512111] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:32 2019][2937734.518579] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:32 2019][2937734.524291] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:32 2019][2937734.531901] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937734.536958] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.543233] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937734.549874] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.556156] osp-pre-19-3 S ffff8b83e63530c0 0 2259 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937734.563482] Call Trace: [Fri Apr 12 11:13:32 2019][2937734.566118] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:32 2019][2937734.571540] [] schedule+0x29/0x70 [Fri Apr 12 11:13:32 2019][2937734.576695] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:32 2019][2937734.583665] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937734.589424] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:32 2019][2937734.596155] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937734.601218] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.607496] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937734.614122] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.620394] osp-syn-18-3 S ffff8b83e6354100 0 2260 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937734.627687] Call Trace: [Fri Apr 12 11:13:32 2019][2937734.630338] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:32 2019][2937734.636467] [] schedule+0x29/0x70 [Fri Apr 12 11:13:32 2019][2937734.641619] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:32 2019][2937734.648955] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937734.654734] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.662059] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:32 2019][2937734.667730] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:32 2019][2937734.674292] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.681543] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.688805] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.696146] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.703912] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.711209] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.718995] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.726678] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:32 2019][2937734.733123] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:32 2019][2937734.739146] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:32 2019][2937734.745704] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:32 2019][2937734.752500] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:32 2019][2937734.758980] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:32 2019][2937734.764650] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:32 2019][2937734.772232] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937734.777291] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.783576] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937734.790213] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.796489] osp-pre-18-3 S ffff8b83e6355140 0 2261 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937734.803780] Call Trace: [Fri Apr 12 11:13:32 2019][2937734.806413] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:32 2019][2937734.811819] [] schedule+0x29/0x70 [Fri Apr 12 11:13:32 2019][2937734.816974] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:32 2019][2937734.823944] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:32 2019][2937734.829702] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:32 2019][2937734.836414] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:32 2019][2937734.841487] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.847779] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:32 2019][2937734.854406] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:32 2019][2937734.860699] osp-syn-25-3 S ffff8b83e6356180 0 2262 2 0x00000080 [Fri Apr 12 11:13:32 2019][2937734.867984] Call Trace: [Fri Apr 12 11:13:32 2019][2937734.870627] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:33 2019][2937734.876740] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937734.881898] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:33 2019][2937734.889225] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937734.895001] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.902334] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:33 2019][2937734.908037] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937734.914597] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.921847] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.929090] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.936463] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.944283] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.951699] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.959472] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.967140] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:33 2019][2937734.973597] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:33 2019][2937734.979624] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937734.986199] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:33 2019][2937734.993008] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:33 2019][2937734.999456] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:33 2019][2937735.005127] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:33 2019][2937735.012702] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:33 2019][2937735.017794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.024083] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:33 2019][2937735.030707] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.036991] osp-pre-25-3 S ffff8b83e57a8000 0 2263 2 0x00000080 [Fri Apr 12 11:13:33 2019][2937735.044266] Call Trace: [Fri Apr 12 11:13:33 2019][2937735.046900] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:33 2019][2937735.052308] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937735.057461] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:33 2019][2937735.064456] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937735.070218] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:33 2019][2937735.076932] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:33 2019][2937735.082023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.088312] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:33 2019][2937735.094940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.101221] osp-syn-17-3 S ffff8b83e57a9040 0 2264 2 0x00000080 [Fri Apr 12 11:13:33 2019][2937735.108508] Call Trace: [Fri Apr 12 11:13:33 2019][2937735.111141] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:33 2019][2937735.117269] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937735.122459] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:33 2019][2937735.129785] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937735.135566] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.142895] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937735.149484] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.156738] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.163984] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.171352] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.179158] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.186419] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.194197] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.201865] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:33 2019][2937735.208323] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:33 2019][2937735.214343] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937735.220930] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.227732] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:33 2019][2937735.234191] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:33 2019][2937735.239885] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:33 2019][2937735.247482] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:33 2019][2937735.252576] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:13:33 2019][2937735.258863] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.265157] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:33 2019][2937735.271781] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.278053] osp-pre-17-3 S ffff8b83e57aa080 0 2265 2 0x00000080 [Fri Apr 12 11:13:33 2019][2937735.285338] Call Trace: [Fri Apr 12 11:13:33 2019][2937735.287973] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:33 2019][2937735.293386] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937735.298540] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:33 2019][2937735.305510] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937735.311272] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:33 2019][2937735.317992] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:33 2019][2937735.323051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.329328] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:33 2019][2937735.335968] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.342242] osp-syn-0-3 S ffff8b83e57ab0c0 0 2266 2 0x00000080 [Fri Apr 12 11:13:33 2019][2937735.349525] Call Trace: [Fri Apr 12 11:13:33 2019][2937735.352166] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:33 2019][2937735.358279] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937735.363456] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:33 2019][2937735.370775] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937735.376550] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.383873] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937735.390431] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.397678] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.404920] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.412252] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.420016] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.427274] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.435049] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.442720] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:33 2019][2937735.449171] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:33 2019][2937735.455190] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937735.461745] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.468548] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:33 2019][2937735.475010] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:33 2019][2937735.480711] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:33 2019][2937735.488313] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:33 2019][2937735.493374] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.499650] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:33 2019][2937735.506281] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.512560] osp-pre-0-3 S ffff8b83e57ac100 0 2267 2 0x00000080 [Fri Apr 12 11:13:33 2019][2937735.519853] Call Trace: [Fri Apr 12 11:13:33 2019][2937735.522495] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:33 2019][2937735.527947] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937735.533100] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:33 2019][2937735.540071] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937735.545828] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:33 2019][2937735.552530] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:33 2019][2937735.557590] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.563860] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:33 2019][2937735.570502] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.576772] osp-syn-5-3 S ffff8b83e57ad140 0 2268 2 0x00000080 [Fri Apr 12 11:13:33 2019][2937735.584048] Call Trace: [Fri Apr 12 11:13:33 2019][2937735.586681] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:33 2019][2937735.592780] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937735.597944] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:33 2019][2937735.605273] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937735.611057] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.618382] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937735.624937] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.632182] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.639428] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.646762] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.654533] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.661791] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.669572] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.677236] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:33 2019][2937735.683689] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:33 2019][2937735.689721] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937735.696287] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.703103] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:33 2019][2937735.709562] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:33 2019][2937735.715235] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:33 2019][2937735.722813] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:33 2019][2937735.727875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.734160] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:33 2019][2937735.740797] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.747069] osp-pre-5-3 S ffff8b83e57ae180 0 2269 2 0x00000080 [Fri Apr 12 11:13:33 2019][2937735.754354] Call Trace: [Fri Apr 12 11:13:33 2019][2937735.756988] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:33 2019][2937735.762411] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937735.767567] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:33 2019][2937735.774555] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937735.780330] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:33 2019][2937735.787052] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:33 2019][2937735.792130] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.798415] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:33 2019][2937735.805052] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:33 2019][2937735.811323] osp-syn-44-3 S ffff8b83e5320000 0 2270 2 0x00000080 [Fri Apr 12 11:13:33 2019][2937735.818600] Call Trace: [Fri Apr 12 11:13:33 2019][2937735.821233] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:33 2019][2937735.827335] [] schedule+0x29/0x70 [Fri Apr 12 11:13:33 2019][2937735.832506] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:33 2019][2937735.839826] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:33 2019][2937735.845600] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.852942] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:33 2019][2937735.858627] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:33 2019][2937735.865188] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:33 2019][2937735.872436] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:34 2019][2937735.879700] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937735.887045] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:34 2019][2937735.894865] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:34 2019][2937735.902115] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:34 2019][2937735.909885] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937735.917551] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:34 2019][2937735.924016] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:34 2019][2937735.930033] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:34 2019][2937735.936596] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:34 2019][2937735.943398] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:34 2019][2937735.949847] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:34 2019][2937735.955532] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:34 2019][2937735.963143] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:34 2019][2937735.968219] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937735.974500] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:34 2019][2937735.981120] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937735.987392] osp-pre-44-3 S ffff8b83e5321040 0 2271 2 0x00000080 [Fri Apr 12 11:13:34 2019][2937735.994670] Call Trace: [Fri Apr 12 11:13:34 2019][2937735.997310] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:34 2019][2937736.002725] [] schedule+0x29/0x70 [Fri Apr 12 11:13:34 2019][2937736.007884] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:34 2019][2937736.014852] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:34 2019][2937736.020619] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:34 2019][2937736.027342] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:34 2019][2937736.032400] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.038674] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:34 2019][2937736.045295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.051570] osp-syn-32-3 S ffff8b83e5322080 0 2272 2 0x00000080 [Fri Apr 12 11:13:34 2019][2937736.058855] Call Trace: [Fri Apr 12 11:13:34 2019][2937736.061488] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:34 2019][2937736.067587] [] schedule+0x29/0x70 [Fri Apr 12 11:13:34 2019][2937736.072750] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:34 2019][2937736.080085] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:34 2019][2937736.085876] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.093195] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:34 2019][2937736.098881] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:34 2019][2937736.105457] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.112711] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.119989] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.128744] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:34 2019][2937736.135849] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.143619] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.150869] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.158645] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.166308] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:34 2019][2937736.172769] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:34 2019][2937736.178797] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:34 2019][2937736.185377] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.192180] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:34 2019][2937736.198650] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:34 2019][2937736.204330] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:34 2019][2937736.211913] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:34 2019][2937736.216982] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.223267] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:34 2019][2937736.229921] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.236203] osp-pre-32-3 S ffff8b83e53230c0 0 2273 2 0x00000080 [Fri Apr 12 11:13:34 2019][2937736.243483] Call Trace: [Fri Apr 12 11:13:34 2019][2937736.246117] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:34 2019][2937736.251524] [] schedule+0x29/0x70 [Fri Apr 12 11:13:34 2019][2937736.256678] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:34 2019][2937736.263654] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:34 2019][2937736.269415] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:34 2019][2937736.276124] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:34 2019][2937736.281184] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.287468] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:34 2019][2937736.294089] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.300364] osp-syn-21-3 S ffff8b83e5324100 0 2274 2 0x00000080 [Fri Apr 12 11:13:34 2019][2937736.307636] Call Trace: [Fri Apr 12 11:13:34 2019][2937736.310296] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:34 2019][2937736.317109] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:34 2019][2937736.323210] [] schedule+0x29/0x70 [Fri Apr 12 11:13:34 2019][2937736.328364] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:34 2019][2937736.335705] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:34 2019][2937736.341491] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.348822] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:34 2019][2937736.354500] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:34 2019][2937736.361059] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.368314] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.375560] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.382888] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:34 2019][2937736.389888] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.397659] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.404905] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.412675] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.420337] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:34 2019][2937736.426788] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:34 2019][2937736.432804] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:34 2019][2937736.439358] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.446162] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:34 2019][2937736.452628] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:34 2019][2937736.458314] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:34 2019][2937736.465902] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:34 2019][2937736.470969] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.477254] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:34 2019][2937736.483876] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.490156] osp-pre-21-3 S ffff8b83e5325140 0 2275 2 0x00000080 [Fri Apr 12 11:13:34 2019][2937736.497460] Call Trace: [Fri Apr 12 11:13:34 2019][2937736.500103] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:34 2019][2937736.505511] [] schedule+0x29/0x70 [Fri Apr 12 11:13:34 2019][2937736.510672] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:34 2019][2937736.517647] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:34 2019][2937736.523409] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:34 2019][2937736.530123] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:34 2019][2937736.535192] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.541485] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:34 2019][2937736.548104] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.554396] osp-syn-16-3 S ffff8b83e5326180 0 2276 2 0x00000080 [Fri Apr 12 11:13:34 2019][2937736.561681] Call Trace: [Fri Apr 12 11:13:34 2019][2937736.564349] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:34 2019][2937736.570479] [] schedule+0x29/0x70 [Fri Apr 12 11:13:34 2019][2937736.575658] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:34 2019][2937736.583012] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:34 2019][2937736.588845] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.596164] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:34 2019][2937736.601834] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:34 2019][2937736.608394] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.615641] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.622918] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.630251] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.638015] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.645260] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.653075] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.660753] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:34 2019][2937736.667207] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:34 2019][2937736.673244] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:34 2019][2937736.679803] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.686613] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:34 2019][2937736.693062] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:34 2019][2937736.698740] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:34 2019][2937736.706317] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:34 2019][2937736.711389] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.717665] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:34 2019][2937736.724295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.730577] osp-pre-16-3 S ffff8b83fdeb0000 0 2277 2 0x00000080 [Fri Apr 12 11:13:34 2019][2937736.737858] Call Trace: [Fri Apr 12 11:13:34 2019][2937736.740492] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:34 2019][2937736.745900] [] schedule+0x29/0x70 [Fri Apr 12 11:13:34 2019][2937736.751052] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:34 2019][2937736.758016] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:34 2019][2937736.763774] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:34 2019][2937736.770478] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:34 2019][2937736.775571] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.782282] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:34 2019][2937736.788967] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:34 2019][2937736.795238] osp-syn-41-3 S ffff8b83fdeb1040 0 2278 2 0x00000080 [Fri Apr 12 11:13:34 2019][2937736.802513] Call Trace: [Fri Apr 12 11:13:34 2019][2937736.805145] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:34 2019][2937736.811248] [] schedule+0x29/0x70 [Fri Apr 12 11:13:34 2019][2937736.816400] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:34 2019][2937736.823708] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:34 2019][2937736.829490] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.836814] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:34 2019][2937736.843379] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.850634] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:34 2019][2937736.857886] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937736.865251] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937736.873035] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937736.880298] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937736.888065] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937736.895731] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:35 2019][2937736.902191] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:35 2019][2937736.908248] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:35 2019][2937736.914819] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:35 2019][2937736.921618] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:35 2019][2937736.928080] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:35 2019][2937736.933753] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:35 2019][2937736.941330] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:35 2019][2937736.946392] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937736.952712] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:35 2019][2937736.959340] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937736.965610] osp-pre-41-3 S ffff8b83fdeb2080 0 2279 2 0x00000080 [Fri Apr 12 11:13:35 2019][2937736.972904] Call Trace: [Fri Apr 12 11:13:35 2019][2937736.975536] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:35 2019][2937736.980963] [] schedule+0x29/0x70 [Fri Apr 12 11:13:35 2019][2937736.986119] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:35 2019][2937736.993090] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:35 2019][2937736.998848] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:35 2019][2937737.005568] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:35 2019][2937737.010628] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.016905] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:35 2019][2937737.023535] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.029807] osp-syn-4-3 S ffff8b83fdeb30c0 0 2280 2 0x00000080 [Fri Apr 12 11:13:35 2019][2937737.037128] Call Trace: [Fri Apr 12 11:13:35 2019][2937737.040297] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:35 2019][2937737.046549] [] schedule+0x29/0x70 [Fri Apr 12 11:13:35 2019][2937737.051708] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:35 2019][2937737.059024] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:35 2019][2937737.064805] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.072135] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:35 2019][2937737.078703] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.085955] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.093201] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.100521] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:35 2019][2937737.107511] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.115301] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.122553] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.130319] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.137981] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:35 2019][2937737.144426] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:35 2019][2937737.150446] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:35 2019][2937737.156996] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.163818] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:35 2019][2937737.170264] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:35 2019][2937737.175937] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:35 2019][2937737.183504] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:35 2019][2937737.188566] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.194841] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:35 2019][2937737.201462] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.207736] osp-pre-4-3 S ffff8b83fdeb4100 0 2281 2 0x00000080 [Fri Apr 12 11:13:35 2019][2937737.215016] Call Trace: [Fri Apr 12 11:13:35 2019][2937737.217669] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:35 2019][2937737.223090] [] schedule+0x29/0x70 [Fri Apr 12 11:13:35 2019][2937737.228264] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:35 2019][2937737.235234] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:35 2019][2937737.240999] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:35 2019][2937737.247709] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:35 2019][2937737.252782] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.259057] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:35 2019][2937737.265692] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.271969] osp-syn-23-3 S ffff8b83fdeb5140 0 2282 2 0x00000080 [Fri Apr 12 11:13:35 2019][2937737.279245] Call Trace: [Fri Apr 12 11:13:35 2019][2937737.281876] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:35 2019][2937737.287977] [] schedule+0x29/0x70 [Fri Apr 12 11:13:35 2019][2937737.293132] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:35 2019][2937737.300469] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:35 2019][2937737.306248] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.313575] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:35 2019][2937737.319252] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:35 2019][2937737.325806] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.333064] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.340317] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.347663] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.355444] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.362717] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.370488] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.378152] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:35 2019][2937737.384605] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:35 2019][2937737.390626] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:35 2019][2937737.397194] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.404007] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:35 2019][2937737.410468] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:35 2019][2937737.416137] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:35 2019][2937737.423722] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:35 2019][2937737.428788] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.435065] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:35 2019][2937737.441686] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.447959] osp-pre-23-3 S ffff8b83fdeb6180 0 2283 2 0x00000080 [Fri Apr 12 11:13:35 2019][2937737.455253] Call Trace: [Fri Apr 12 11:13:35 2019][2937737.457894] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:35 2019][2937737.463297] [] schedule+0x29/0x70 [Fri Apr 12 11:13:35 2019][2937737.468452] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:35 2019][2937737.475435] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:35 2019][2937737.481201] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:35 2019][2937737.487915] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:35 2019][2937737.492983] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.499268] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:35 2019][2937737.505889] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.512164] osp-syn-33-3 S ffff8b8c4f5d8000 0 2284 2 0x00000080 [Fri Apr 12 11:13:35 2019][2937737.519437] Call Trace: [Fri Apr 12 11:13:35 2019][2937737.522071] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:35 2019][2937737.528303] [] schedule+0x29/0x70 [Fri Apr 12 11:13:35 2019][2937737.533487] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:35 2019][2937737.540820] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:35 2019][2937737.546595] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.553929] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:35 2019][2937737.560495] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.567760] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.575028] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.582373] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:35 2019][2937737.589387] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.597160] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.604426] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.612197] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.619862] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:35 2019][2937737.626309] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:35 2019][2937737.632324] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:35 2019][2937737.638879] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.645682] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:35 2019][2937737.652136] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:35 2019][2937737.657807] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:35 2019][2937737.665383] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:35 2019][2937737.670463] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.676741] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:35 2019][2937737.683373] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.689652] osp-pre-33-3 S ffff8b8c4f5d9040 0 2285 2 0x00000080 [Fri Apr 12 11:13:35 2019][2937737.696926] Call Trace: [Fri Apr 12 11:13:35 2019][2937737.699563] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:35 2019][2937737.704978] [] schedule+0x29/0x70 [Fri Apr 12 11:13:35 2019][2937737.710164] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:35 2019][2937737.717153] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:35 2019][2937737.722919] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:35 2019][2937737.729816] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:35 2019][2937737.734875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.741153] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:35 2019][2937737.747770] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:35 2019][2937737.754045] osp-syn-20-3 S ffff8b8c4f5da080 0 2286 2 0x00000080 [Fri Apr 12 11:13:35 2019][2937737.761322] Call Trace: [Fri Apr 12 11:13:35 2019][2937737.763952] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:35 2019][2937737.770052] [] schedule+0x29/0x70 [Fri Apr 12 11:13:35 2019][2937737.775211] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:35 2019][2937737.782539] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:35 2019][2937737.788342] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.795677] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:35 2019][2937737.801349] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:35 2019][2937737.807936] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.815195] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.822466] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.829810] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:35 2019][2937737.836802] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.844563] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.851821] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:35 2019][2937737.859636] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937737.867310] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:36 2019][2937737.873763] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:36 2019][2937737.879781] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937737.886333] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:36 2019][2937737.893139] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:36 2019][2937737.899590] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:36 2019][2937737.905262] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:36 2019][2937737.912833] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:36 2019][2937737.917903] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937737.924179] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:36 2019][2937737.930798] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937737.937073] osp-pre-20-3 S ffff8b8c4f5db0c0 0 2287 2 0x00000080 [Fri Apr 12 11:13:36 2019][2937737.944366] Call Trace: [Fri Apr 12 11:13:36 2019][2937737.947010] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:36 2019][2937737.952413] [] schedule+0x29/0x70 [Fri Apr 12 11:13:36 2019][2937737.957571] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:36 2019][2937737.964539] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:36 2019][2937737.970301] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:36 2019][2937737.977008] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:36 2019][2937737.982101] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937737.988374] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:36 2019][2937737.994995] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.001270] osp-syn-7-3 S ffff8b8c4f5dc100 0 2288 2 0x00000080 [Fri Apr 12 11:13:36 2019][2937738.008561] Call Trace: [Fri Apr 12 11:13:36 2019][2937738.011202] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:36 2019][2937738.017994] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:36 2019][2937738.024096] [] schedule+0x29/0x70 [Fri Apr 12 11:13:36 2019][2937738.029265] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:36 2019][2937738.036606] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:36 2019][2937738.042391] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.049718] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:36 2019][2937738.055391] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937738.061947] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.069195] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.076446] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.083777] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:36 2019][2937738.090765] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.098528] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.105795] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.113566] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.121239] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:36 2019][2937738.127683] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:36 2019][2937738.133706] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937738.140265] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.147068] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:36 2019][2937738.153519] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:36 2019][2937738.159202] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:36 2019][2937738.166792] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:36 2019][2937738.171851] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.178128] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:36 2019][2937738.184758] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.191066] osp-pre-7-3 S ffff8b8c4f5dd140 0 2289 2 0x00000080 [Fri Apr 12 11:13:36 2019][2937738.198340] Call Trace: [Fri Apr 12 11:13:36 2019][2937738.200977] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:36 2019][2937738.206485] [] schedule+0x29/0x70 [Fri Apr 12 11:13:36 2019][2937738.211641] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:36 2019][2937738.218611] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:36 2019][2937738.224367] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:36 2019][2937738.231078] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:36 2019][2937738.236140] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.242425] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:36 2019][2937738.249046] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.255318] osp-syn-40-3 S ffff8b8c4f5de180 0 2290 2 0x00000080 [Fri Apr 12 11:13:36 2019][2937738.262597] Call Trace: [Fri Apr 12 11:13:36 2019][2937738.265241] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:36 2019][2937738.271354] [] schedule+0x29/0x70 [Fri Apr 12 11:13:36 2019][2937738.276510] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:36 2019][2937738.283838] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:36 2019][2937738.289614] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.296953] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:36 2019][2937738.302632] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937738.309199] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.316454] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.323713] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.331046] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.338820] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.346092] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.353855] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.361519] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:36 2019][2937738.367994] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:36 2019][2937738.374012] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937738.380569] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.387366] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:36 2019][2937738.393821] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:36 2019][2937738.399496] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:36 2019][2937738.407092] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:36 2019][2937738.412157] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.418433] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:36 2019][2937738.425113] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.431396] osp-pre-40-3 S ffff8b890ef98000 0 2291 2 0x00000080 [Fri Apr 12 11:13:36 2019][2937738.438735] Call Trace: [Fri Apr 12 11:13:36 2019][2937738.441377] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:36 2019][2937738.446801] [] schedule+0x29/0x70 [Fri Apr 12 11:13:36 2019][2937738.451952] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:36 2019][2937738.458922] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:36 2019][2937738.464680] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:36 2019][2937738.471385] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:36 2019][2937738.476450] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.482724] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:36 2019][2937738.489350] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.495625] osp-syn-45-3 S ffff8b890ef99040 0 2292 2 0x00000080 [Fri Apr 12 11:13:36 2019][2937738.502903] Call Trace: [Fri Apr 12 11:13:36 2019][2937738.505537] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:36 2019][2937738.511663] [] schedule+0x29/0x70 [Fri Apr 12 11:13:36 2019][2937738.516847] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:36 2019][2937738.524177] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:36 2019][2937738.529952] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.537269] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:36 2019][2937738.542942] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937738.549524] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.556774] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.564019] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.571349] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.579126] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.586368] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.594135] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.601798] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:36 2019][2937738.608244] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:36 2019][2937738.614263] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937738.620812] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.627609] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:36 2019][2937738.634053] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:36 2019][2937738.639728] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:36 2019][2937738.647308] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:36 2019][2937738.652374] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.658663] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:36 2019][2937738.665284] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.671563] osp-pre-45-3 S ffff8b890ef9a080 0 2293 2 0x00000080 [Fri Apr 12 11:13:36 2019][2937738.678866] Call Trace: [Fri Apr 12 11:13:36 2019][2937738.681507] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:36 2019][2937738.686913] [] schedule+0x29/0x70 [Fri Apr 12 11:13:36 2019][2937738.692068] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:36 2019][2937738.699081] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:36 2019][2937738.704859] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:36 2019][2937738.711577] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:36 2019][2937738.716640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.722913] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:36 2019][2937738.729534] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:36 2019][2937738.735807] osp-syn-6-3 S ffff8b890ef9b0c0 0 2294 2 0x00000080 [Fri Apr 12 11:13:36 2019][2937738.743096] Call Trace: [Fri Apr 12 11:13:36 2019][2937738.745725] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:36 2019][2937738.751828] [] schedule+0x29/0x70 [Fri Apr 12 11:13:36 2019][2937738.756979] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:36 2019][2937738.764302] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:36 2019][2937738.770087] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.777415] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937738.783977] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.791223] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.798492] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.805856] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:36 2019][2937738.812857] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.820637] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.827882] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.835654] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:36 2019][2937738.843318] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:36 2019][2937738.849762] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:36 2019][2937738.855796] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:36 2019][2937738.862361] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:37 2019][2937738.869168] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:37 2019][2937738.875630] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:37 2019][2937738.881307] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:37 2019][2937738.888885] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937738.893946] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937738.900235] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937738.906852] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937738.913125] osp-pre-6-3 S ffff8b890ef9c100 0 2295 2 0x00000080 [Fri Apr 12 11:13:37 2019][2937738.920402] Call Trace: [Fri Apr 12 11:13:37 2019][2937738.923033] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:37 2019][2937738.928454] [] schedule+0x29/0x70 [Fri Apr 12 11:13:37 2019][2937738.933636] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:37 2019][2937738.940621] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:37 2019][2937738.946377] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:37 2019][2937738.953094] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937738.958150] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937738.964428] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937738.971049] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937738.977319] osp-syn-24-3 S ffff8b890ef9d140 0 2296 2 0x00000080 [Fri Apr 12 11:13:37 2019][2937738.984599] Call Trace: [Fri Apr 12 11:13:37 2019][2937738.987257] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:37 2019][2937738.994059] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:37 2019][2937739.000158] [] schedule+0x29/0x70 [Fri Apr 12 11:13:37 2019][2937739.005313] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:37 2019][2937739.012630] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:37 2019][2937739.018406] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.025883] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:37 2019][2937739.031570] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:37 2019][2937739.038166] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.045414] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.052656] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.059990] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.067765] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.075027] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.082808] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.090495] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:37 2019][2937739.096944] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:37 2019][2937739.102973] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:37 2019][2937739.109533] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.116352] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:37 2019][2937739.122807] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:37 2019][2937739.128479] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:37 2019][2937739.136056] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937739.141127] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.147432] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937739.154057] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.160337] osp-pre-24-3 S ffff8b890ef9e180 0 2297 2 0x00000080 [Fri Apr 12 11:13:37 2019][2937739.167625] Call Trace: [Fri Apr 12 11:13:37 2019][2937739.170255] [] schedule+0x29/0x70 [Fri Apr 12 11:13:37 2019][2937739.175411] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:37 2019][2937739.182384] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:37 2019][2937739.188169] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:37 2019][2937739.194886] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937739.199957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.206242] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937739.212867] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.219144] osp-syn-47-3 S ffff8b8cb46c0000 0 2298 2 0x00000080 [Fri Apr 12 11:13:37 2019][2937739.226439] Call Trace: [Fri Apr 12 11:13:37 2019][2937739.229079] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:37 2019][2937739.235191] [] schedule+0x29/0x70 [Fri Apr 12 11:13:37 2019][2937739.240342] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:37 2019][2937739.247681] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:37 2019][2937739.253453] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.260764] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:37 2019][2937739.266440] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:37 2019][2937739.272999] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.280250] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.287510] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.294855] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.302624] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.309881] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.317647] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.325319] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:37 2019][2937739.331765] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:37 2019][2937739.337796] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:37 2019][2937739.344377] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.351184] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:37 2019][2937739.357636] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:37 2019][2937739.363308] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:37 2019][2937739.370895] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937739.375954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.382232] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937739.388855] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.395135] osp-pre-47-3 S ffff8b8cb46c1040 0 2299 2 0x00000080 [Fri Apr 12 11:13:37 2019][2937739.402412] Call Trace: [Fri Apr 12 11:13:37 2019][2937739.405052] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:37 2019][2937739.410459] [] schedule+0x29/0x70 [Fri Apr 12 11:13:37 2019][2937739.415614] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:37 2019][2937739.422595] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:37 2019][2937739.428361] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:37 2019][2937739.435084] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937739.440141] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.446417] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937739.453048] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.459318] osp-syn-38-3 S ffff8b8cb46c2080 0 2300 2 0x00000080 [Fri Apr 12 11:13:37 2019][2937739.466635] Call Trace: [Fri Apr 12 11:13:37 2019][2937739.469277] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:37 2019][2937739.475390] [] schedule+0x29/0x70 [Fri Apr 12 11:13:37 2019][2937739.480545] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:37 2019][2937739.487863] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:37 2019][2937739.493639] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.500963] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:37 2019][2937739.507518] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.514768] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.522011] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.529347] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.537115] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.544363] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.552129] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.559791] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:37 2019][2937739.566238] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:37 2019][2937739.572311] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:37 2019][2937739.578877] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.585692] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:37 2019][2937739.592142] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:37 2019][2937739.597811] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:37 2019][2937739.605382] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937739.610443] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.616719] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937739.623338] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.629604] osp-pre-38-3 S ffff8b8cb46c30c0 0 2301 2 0x00000080 [Fri Apr 12 11:13:37 2019][2937739.636891] Call Trace: [Fri Apr 12 11:13:37 2019][2937739.639523] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:37 2019][2937739.644928] [] schedule+0x29/0x70 [Fri Apr 12 11:13:37 2019][2937739.650084] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:37 2019][2937739.657047] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:37 2019][2937739.662804] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:37 2019][2937739.669509] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937739.674568] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.680834] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937739.687450] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.693722] osp-syn-15-3 S ffff8b8cb46c4100 0 2302 2 0x00000080 [Fri Apr 12 11:13:37 2019][2937739.700990] Call Trace: [Fri Apr 12 11:13:37 2019][2937739.703619] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:37 2019][2937739.709714] [] schedule+0x29/0x70 [Fri Apr 12 11:13:37 2019][2937739.714868] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:37 2019][2937739.722176] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:37 2019][2937739.727953] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.735271] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:37 2019][2937739.741827] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.749075] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.756328] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.763651] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:37 2019][2937739.770639] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.778405] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.785667] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.793430] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.801095] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:37 2019][2937739.807570] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:37 2019][2937739.813601] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:37 2019][2937739.820185] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:37 2019][2937739.827006] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:37 2019][2937739.833464] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:37 2019][2937739.839142] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:37 2019][2937739.846716] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:37 2019][2937739.851773] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:37 2019][2937739.858049] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:37 2019][2937739.864669] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937739.870941] osp-pre-15-3 S ffff8b8cb46c5140 0 2303 2 0x00000080 [Fri Apr 12 11:13:38 2019][2937739.878212] Call Trace: [Fri Apr 12 11:13:38 2019][2937739.880843] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:38 2019][2937739.886251] [] schedule+0x29/0x70 [Fri Apr 12 11:13:38 2019][2937739.891409] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:38 2019][2937739.898408] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:38 2019][2937739.904172] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:38 2019][2937739.910885] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:38 2019][2937739.915945] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937739.922217] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:38 2019][2937739.928851] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937739.935156] osp-syn-29-3 S ffff8b8cb46c6180 0 2304 2 0x00000080 [Fri Apr 12 11:13:38 2019][2937739.942444] Call Trace: [Fri Apr 12 11:13:38 2019][2937739.945101] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:38 2019][2937739.951202] [] schedule+0x29/0x70 [Fri Apr 12 11:13:38 2019][2937739.956353] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:38 2019][2937739.963664] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:38 2019][2937739.969440] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937739.976756] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:38 2019][2937739.983322] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:38 2019][2937739.990568] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:38 2019][2937739.997813] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.005161] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:38 2019][2937740.012160] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.019932] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.027176] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.034951] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.042616] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:38 2019][2937740.049070] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:38 2019][2937740.055089] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:38 2019][2937740.061658] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.068480] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:38 2019][2937740.074942] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:38 2019][2937740.080611] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:38 2019][2937740.088212] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:38 2019][2937740.093419] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.099693] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:38 2019][2937740.106322] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.112595] osp-pre-29-3 S ffff8b8b95120000 0 2305 2 0x00000080 [Fri Apr 12 11:13:38 2019][2937740.119888] Call Trace: [Fri Apr 12 11:13:38 2019][2937740.122525] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:38 2019][2937740.127963] [] schedule+0x29/0x70 [Fri Apr 12 11:13:38 2019][2937740.133117] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:38 2019][2937740.140136] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:38 2019][2937740.145897] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:38 2019][2937740.152612] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:38 2019][2937740.157671] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:13:38 2019][2937740.163947] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.170222] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:38 2019][2937740.176856] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.183315] osp-syn-1-3 S ffff8b8b95121040 0 2306 2 0x00000080 [Fri Apr 12 11:13:38 2019][2937740.190602] Call Trace: [Fri Apr 12 11:13:38 2019][2937740.193234] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:38 2019][2937740.199344] [] schedule+0x29/0x70 [Fri Apr 12 11:13:38 2019][2937740.204524] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:38 2019][2937740.211843] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:38 2019][2937740.217621] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.224944] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:38 2019][2937740.231499] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.238755] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.246010] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.253349] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.261122] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.268366] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.276131] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.283817] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:38 2019][2937740.290277] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:38 2019][2937740.296296] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:38 2019][2937740.302877] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.309675] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:38 2019][2937740.316125] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:38 2019][2937740.321808] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:38 2019][2937740.329380] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:38 2019][2937740.334437] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.340716] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:38 2019][2937740.347339] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.353609] osp-pre-1-3 S ffff8b8b95122080 0 2307 2 0x00000080 [Fri Apr 12 11:13:38 2019][2937740.360914] Call Trace: [Fri Apr 12 11:13:38 2019][2937740.363557] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:38 2019][2937740.368962] [] schedule+0x29/0x70 [Fri Apr 12 11:13:38 2019][2937740.374117] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:38 2019][2937740.381110] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:38 2019][2937740.386879] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:38 2019][2937740.393615] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:38 2019][2937740.398690] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.404964] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:38 2019][2937740.411582] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.417871] osp-syn-22-3 S ffff8b8b951230c0 0 2308 2 0x00000080 [Fri Apr 12 11:13:38 2019][2937740.425144] Call Trace: [Fri Apr 12 11:13:38 2019][2937740.427785] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:38 2019][2937740.434582] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:38 2019][2937740.440689] [] schedule+0x29/0x70 [Fri Apr 12 11:13:38 2019][2937740.445854] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:38 2019][2937740.453208] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:38 2019][2937740.458996] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.466317] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:38 2019][2937740.471990] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:38 2019][2937740.478545] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.485792] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.493036] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.500355] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:38 2019][2937740.507349] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.515110] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.522357] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.530120] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.537786] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:38 2019][2937740.544235] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:38 2019][2937740.550270] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:38 2019][2937740.556829] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.563634] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:38 2019][2937740.570082] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:38 2019][2937740.575755] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:38 2019][2937740.583327] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:38 2019][2937740.588425] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.594718] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:38 2019][2937740.601367] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.607654] osp-pre-22-3 S ffff8b8b95124100 0 2309 2 0x00000080 [Fri Apr 12 11:13:38 2019][2937740.614949] Call Trace: [Fri Apr 12 11:13:38 2019][2937740.617592] [] schedule+0x29/0x70 [Fri Apr 12 11:13:38 2019][2937740.622752] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:38 2019][2937740.629744] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:38 2019][2937740.635523] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:38 2019][2937740.642248] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:38 2019][2937740.647316] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.653593] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:38 2019][2937740.660213] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.666493] osp-syn-3-3 S ffff8b8b95125140 0 2310 2 0x00000080 [Fri Apr 12 11:13:38 2019][2937740.673771] Call Trace: [Fri Apr 12 11:13:38 2019][2937740.676412] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:38 2019][2937740.682511] [] schedule+0x29/0x70 [Fri Apr 12 11:13:38 2019][2937740.687667] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:38 2019][2937740.694984] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:38 2019][2937740.700762] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.708106] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:38 2019][2937740.713789] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:38 2019][2937740.720345] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.727587] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.734832] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.742152] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:38 2019][2937740.749133] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.756924] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.764180] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.771947] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.779612] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:38 2019][2937740.786064] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:38 2019][2937740.792084] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:38 2019][2937740.798644] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:38 2019][2937740.805449] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:38 2019][2937740.811988] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:38 2019][2937740.817668] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:38 2019][2937740.825249] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:38 2019][2937740.830308] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:38 2019][2937740.836594] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:38 2019][2937740.843230] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937740.849505] osp-pre-3-3 S ffff8b8b95126180 0 2311 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937740.856787] Call Trace: [Fri Apr 12 11:13:39 2019][2937740.859424] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:39 2019][2937740.864829] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937740.869983] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:39 2019][2937740.876954] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:39 2019][2937740.882712] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:39 2019][2937740.889444] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:39 2019][2937740.894503] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937740.900781] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:39 2019][2937740.907400] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937740.913691] osp-syn-27-3 S ffff8b885e750000 0 2312 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937740.920966] Call Trace: [Fri Apr 12 11:13:39 2019][2937740.923599] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:39 2019][2937740.929701] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937740.934851] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:39 2019][2937740.942164] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:39 2019][2937740.947949] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937740.955269] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:39 2019][2937740.960959] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:39 2019][2937740.967512] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:39 2019][2937740.974759] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:39 2019][2937740.982012] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937740.989335] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:39 2019][2937740.996322] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.004090] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.011335] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.019101] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.026763] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:39 2019][2937741.033217] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:39 2019][2937741.039245] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:39 2019][2937741.045813] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.052618] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:39 2019][2937741.059084] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:39 2019][2937741.064762] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:39 2019][2937741.072342] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:39 2019][2937741.077400] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.083679] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:39 2019][2937741.090343] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.096632] osp-pre-27-3 S ffff8b885e751040 0 2313 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937741.104175] Call Trace: [Fri Apr 12 11:13:39 2019][2937741.107559] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:39 2019][2937741.113441] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937741.118715] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:39 2019][2937741.125686] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:39 2019][2937741.131471] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:39 2019][2937741.138189] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:39 2019][2937741.143259] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.149562] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:39 2019][2937741.156184] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.162475] osp-syn-37-3 S ffff8b885e752080 0 2314 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937741.169749] Call Trace: [Fri Apr 12 11:13:39 2019][2937741.172402] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:39 2019][2937741.179202] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:39 2019][2937741.185303] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937741.190458] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:39 2019][2937741.197776] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:39 2019][2937741.203551] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.210889] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:39 2019][2937741.216560] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:39 2019][2937741.223121] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.230382] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.237628] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.244968] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.252733] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.259987] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.267766] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.270898] LNet: Service thread pid 117471 was inactive for 200.11s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [Fri Apr 12 11:13:39 2019][2937741.270900] LNet: Skipped 18 previous similar messages [Fri Apr 12 11:13:39 2019][2937741.270902] LustreError: dumping log to /tmp/lustre-log.1555092819.117471 [Fri Apr 12 11:13:39 2019][2937741.300850] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:39 2019][2937741.307306] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:39 2019][2937741.313329] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:39 2019][2937741.319889] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.326689] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:39 2019][2937741.333146] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:39 2019][2937741.338816] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:39 2019][2937741.346397] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:39 2019][2937741.351458] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.357751] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:39 2019][2937741.364366] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.370642] osp-pre-37-3 S ffff8b885e7530c0 0 2315 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937741.377941] Call Trace: [Fri Apr 12 11:13:39 2019][2937741.380579] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937741.385731] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:39 2019][2937741.392702] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:39 2019][2937741.398461] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:39 2019][2937741.405174] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:39 2019][2937741.410230] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.416509] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:39 2019][2937741.423139] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.429413] osp-syn-36-3 S ffff8b885e754100 0 2316 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937741.436688] Call Trace: [Fri Apr 12 11:13:39 2019][2937741.439321] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:39 2019][2937741.445421] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937741.450573] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:39 2019][2937741.457883] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:39 2019][2937741.463664] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.471006] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:39 2019][2937741.477613] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.484856] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.492100] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.499433] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:39 2019][2937741.506421] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.514195] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.521441] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.536872] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:39 2019][2937741.543322] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:39 2019][2937741.549343] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:39 2019][2937741.555900] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.562692] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:39 2019][2937741.569141] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:39 2019][2937741.574812] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:39 2019][2937741.582381] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:39 2019][2937741.587441] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.593718] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:39 2019][2937741.600339] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.606614] osp-pre-36-3 S ffff8b885e755140 0 2317 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937741.613915] Call Trace: [Fri Apr 12 11:13:39 2019][2937741.616549] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:39 2019][2937741.621956] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937741.627108] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:39 2019][2937741.634074] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:39 2019][2937741.639832] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:39 2019][2937741.646554] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:39 2019][2937741.651617] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.657909] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:39 2019][2937741.664528] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.670821] osp-syn-8-3 S ffff8b885e756180 0 2318 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937741.678095] Call Trace: [Fri Apr 12 11:13:39 2019][2937741.680739] [] ? osd_ldiskfs_read+0x131/0x2d0 [osd_ldiskfs] [Fri Apr 12 11:13:39 2019][2937741.688146] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:39 2019][2937741.694247] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937741.699401] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:39 2019][2937741.706716] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:39 2019][2937741.712498] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.719819] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:39 2019][2937741.726374] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.733621] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.740865] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.748200] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.755962] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.763208] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.770973] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.778635] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:39 2019][2937741.785081] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:39 2019][2937741.791103] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:39 2019][2937741.797651] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:39 2019][2937741.804469] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:39 2019][2937741.810921] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:39 2019][2937741.816591] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:39 2019][2937741.824173] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:39 2019][2937741.829232] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.835518] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:39 2019][2937741.842141] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:39 2019][2937741.848413] osp-pre-8-3 S ffff8b88abb30000 0 2319 2 0x00000080 [Fri Apr 12 11:13:39 2019][2937741.855696] Call Trace: [Fri Apr 12 11:13:39 2019][2937741.858332] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:39 2019][2937741.863745] [] schedule+0x29/0x70 [Fri Apr 12 11:13:39 2019][2937741.868930] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:39 2019][2937741.875904] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937741.881677] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:40 2019][2937741.888384] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:40 2019][2937741.893448] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937741.899721] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:40 2019][2937741.906343] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937741.912618] osp-syn-35-3 S ffff8b88abb31040 0 2320 2 0x00000080 [Fri Apr 12 11:13:40 2019][2937741.919908] Call Trace: [Fri Apr 12 11:13:40 2019][2937741.922539] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:40 2019][2937741.928644] [] schedule+0x29/0x70 [Fri Apr 12 11:13:40 2019][2937741.933796] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:40 2019][2937741.941113] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937741.946913] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937741.954238] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:40 2019][2937741.960800] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:40 2019][2937741.968074] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:40 2019][2937741.975338] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937741.982663] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:40 2019][2937741.989661] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:40 2019][2937741.997433] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.004677] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.012505] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.020204] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:40 2019][2937742.026656] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:40 2019][2937742.032686] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:40 2019][2937742.039245] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.046048] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:40 2019][2937742.052503] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:40 2019][2937742.058173] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:40 2019][2937742.065752] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:40 2019][2937742.070814] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.077098] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:40 2019][2937742.083718] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.089992] osp-pre-35-3 S ffff8b88abb32080 0 2321 2 0x00000080 [Fri Apr 12 11:13:40 2019][2937742.097267] Call Trace: [Fri Apr 12 11:13:40 2019][2937742.099902] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:40 2019][2937742.105306] [] schedule+0x29/0x70 [Fri Apr 12 11:13:40 2019][2937742.110467] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:40 2019][2937742.117464] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937742.123227] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:40 2019][2937742.129938] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:40 2019][2937742.134997] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.141273] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:40 2019][2937742.147917] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.154196] osp-syn-28-3 S ffff8b88abb330c0 0 2322 2 0x00000080 [Fri Apr 12 11:13:40 2019][2937742.161472] Call Trace: [Fri Apr 12 11:13:40 2019][2937742.164104] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:40 2019][2937742.170203] [] schedule+0x29/0x70 [Fri Apr 12 11:13:40 2019][2937742.175375] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:40 2019][2937742.182683] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937742.188461] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.195772] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:40 2019][2937742.201443] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:40 2019][2937742.207993] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.215240] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.222482] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.229803] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:40 2019][2937742.236794] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.244571] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.251810] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.259578] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.267237] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:40 2019][2937742.273684] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:40 2019][2937742.279706] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:40 2019][2937742.286256] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.293053] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:40 2019][2937742.299505] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:40 2019][2937742.305177] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:40 2019][2937742.312761] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:40 2019][2937742.317821] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.324095] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:40 2019][2937742.330735] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.337004] osp-pre-28-3 S ffff8b88abb34100 0 2323 2 0x00000080 [Fri Apr 12 11:13:40 2019][2937742.344294] Call Trace: [Fri Apr 12 11:13:40 2019][2937742.346942] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:40 2019][2937742.352358] [] schedule+0x29/0x70 [Fri Apr 12 11:13:40 2019][2937742.357525] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:40 2019][2937742.364503] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937742.370280] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:40 2019][2937742.376989] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:40 2019][2937742.382051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.388337] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:40 2019][2937742.394981] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.401263] osp-syn-30-3 S ffff8b88abb35140 0 2324 2 0x00000080 [Fri Apr 12 11:13:40 2019][2937742.408558] Call Trace: [Fri Apr 12 11:13:40 2019][2937742.411234] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:40 2019][2937742.418035] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:40 2019][2937742.424134] [] schedule+0x29/0x70 [Fri Apr 12 11:13:40 2019][2937742.429302] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:40 2019][2937742.436627] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937742.442425] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.449741] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:40 2019][2937742.455424] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:40 2019][2937742.462010] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.469298] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.476551] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.483926] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.491692] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.498940] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.506704] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.514367] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:40 2019][2937742.520813] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:40 2019][2937742.526833] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:40 2019][2937742.533383] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.540184] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:40 2019][2937742.546635] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:40 2019][2937742.552304] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:40 2019][2937742.559896] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:40 2019][2937742.564954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.571247] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:40 2019][2937742.577865] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.584141] osp-pre-30-3 S ffff8b88abb36180 0 2325 2 0x00000080 [Fri Apr 12 11:13:40 2019][2937742.591429] Call Trace: [Fri Apr 12 11:13:40 2019][2937742.594068] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:40 2019][2937742.599490] [] schedule+0x29/0x70 [Fri Apr 12 11:13:40 2019][2937742.604649] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:40 2019][2937742.611627] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937742.617418] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:40 2019][2937742.624144] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:40 2019][2937742.629230] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.635504] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:40 2019][2937742.642132] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.648406] osp-syn-34-3 S ffff8b8608ba0000 0 2326 2 0x00000080 [Fri Apr 12 11:13:40 2019][2937742.655717] Call Trace: [Fri Apr 12 11:13:40 2019][2937742.658352] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:40 2019][2937742.664453] [] schedule+0x29/0x70 [Fri Apr 12 11:13:40 2019][2937742.669606] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:40 2019][2937742.676924] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937742.682695] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.690011] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:40 2019][2937742.696560] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.703807] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.711052] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.718398] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.726178] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.733424] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.741188] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.748849] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:40 2019][2937742.755315] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:40 2019][2937742.761334] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:40 2019][2937742.767890] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:40 2019][2937742.774687] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:40 2019][2937742.781133] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:40 2019][2937742.786802] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:40 2019][2937742.794375] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:40 2019][2937742.799444] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.805718] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:40 2019][2937742.812339] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:40 2019][2937742.818633] osp-pre-34-3 S ffff8b8608ba1040 0 2327 2 0x00000080 [Fri Apr 12 11:13:40 2019][2937742.825908] Call Trace: [Fri Apr 12 11:13:40 2019][2937742.828547] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:40 2019][2937742.833957] [] schedule+0x29/0x70 [Fri Apr 12 11:13:40 2019][2937742.839110] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:40 2019][2937742.846082] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:40 2019][2937742.851838] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:41 2019][2937742.858545] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937742.863616] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937742.869899] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937742.876520] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937742.882792] lquota_wb_fir-M S ffff8b89d9ffe180 0 2345 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937742.890078] Call Trace: [Fri Apr 12 11:13:41 2019][2937742.892708] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937742.897860] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:13:41 2019][2937742.903884] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:13:41 2019][2937742.910442] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Fri Apr 12 11:13:41 2019][2937742.917242] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Fri Apr 12 11:13:41 2019][2937742.923867] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937742.929625] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Fri Apr 12 11:13:41 2019][2937742.935984] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937742.941045] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937742.947322] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937742.953951] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937742.960221] lquota_wb_fir-M S ffff8b8c19b82080 0 2348 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937742.967498] Call Trace: [Fri Apr 12 11:13:41 2019][2937742.970130] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937742.975276] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:13:41 2019][2937742.981292] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:13:41 2019][2937742.987834] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Fri Apr 12 11:13:41 2019][2937742.994636] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Fri Apr 12 11:13:41 2019][2937743.001254] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.007012] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Fri Apr 12 11:13:41 2019][2937743.013368] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.018432] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.024715] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.031336] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.037628] dist_txn-3 S ffff8b8c19b830c0 0 2349 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.044910] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.047562] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [Fri Apr 12 11:13:41 2019][2937743.054359] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:41 2019][2937743.060504] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.065725] [] distribute_txn_commit_thread+0x9ac/0xca0 [ptlrpc] [Fri Apr 12 11:13:41 2019][2937743.073563] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.079367] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [Fri Apr 12 11:13:41 2019][2937743.086432] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.091492] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.097769] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.104398] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.110671] jbd2/dm-4-8 S ffff8b5cca7ab0c0 0 2369 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.117965] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.120596] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.125764] [] kjournald2+0x203/0x260 [jbd2] [Fri Apr 12 11:13:41 2019][2937743.131860] [] ? wake_up_atomic_t+0x30/0x30 [Fri Apr 12 11:13:41 2019][2937743.137882] [] ? commit_timeout+0x10/0x10 [jbd2] [Fri Apr 12 11:13:41 2019][2937743.144325] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.149409] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.155690] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.162311] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.168580] ldiskfs-rsv-con S ffff8b8b70a49040 0 2370 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.175884] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.178515] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.183678] [] rescuer_thread+0x31e/0x3c0 [Fri Apr 12 11:13:41 2019][2937743.189532] [] ? process_one_work+0x440/0x440 [Fri Apr 12 11:13:41 2019][2937743.195720] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.200782] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.207066] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.213703] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.219985] osp_up2-1 S ffff8b8b70a4b0c0 0 2381 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.227279] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.229960] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Fri Apr 12 11:13:41 2019][2937743.237024] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:13:41 2019][2937743.243995] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.249149] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:13:41 2019][2937743.256207] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.261968] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:13:41 2019][2937743.269200] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.274264] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.280542] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.287164] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.293436] osp-pre-2-1 S ffff8b87966f4100 0 2382 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.300713] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.303346] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:41 2019][2937743.308751] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.313906] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:41 2019][2937743.320868] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.326628] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:41 2019][2937743.333330] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.338393] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.344667] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.351295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.357571] osp_up0-1 S ffff8b87966f0000 0 2384 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.364857] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.367543] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Fri Apr 12 11:13:41 2019][2937743.374605] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:13:41 2019][2937743.381573] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.386726] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:13:41 2019][2937743.393795] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.399552] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:13:41 2019][2937743.406787] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.411849] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.418122] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.424752] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.431031] osp-pre-0-1 S ffff8b87966f6180 0 2385 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.438308] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.440939] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:41 2019][2937743.446350] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.451505] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:41 2019][2937743.458503] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.464259] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:41 2019][2937743.470991] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.476048] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.482328] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.488954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.495246] osp_up3-1 S ffff8b87966f1040 0 2388 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.502530] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.505210] [] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] [Fri Apr 12 11:13:41 2019][2937743.512277] [] ? osp_get_next_request+0x2d/0x1a0 [osp] [Fri Apr 12 11:13:41 2019][2937743.519244] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.524410] [] osp_send_update_thread+0x2f8/0x590 [osp] [Fri Apr 12 11:13:41 2019][2937743.531513] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.537278] [] ? osp_invalidate_request+0x390/0x390 [osp] [Fri Apr 12 11:13:41 2019][2937743.544512] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.549573] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.555853] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.562499] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.568777] osp-pre-3-1 S ffff8b87966f30c0 0 2389 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.576051] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.578685] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:41 2019][2937743.584094] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.589247] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:41 2019][2937743.596220] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.601985] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:41 2019][2937743.608698] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.613765] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.620042] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.626664] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.632936] osp-syn-12-1 S ffff8b87966f5140 0 2390 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.640212] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.642845] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:41 2019][2937743.648967] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.654136] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:41 2019][2937743.661452] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.667255] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.674580] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:41 2019][2937743.680259] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:41 2019][2937743.686817] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.694081] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.701330] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.708660] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.716425] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.723671] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.731437] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.739098] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:41 2019][2937743.745542] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:41 2019][2937743.751557] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:41 2019][2937743.758107] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:41 2019][2937743.764905] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:41 2019][2937743.771361] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:41 2019][2937743.777038] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:41 2019][2937743.784619] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.789696] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.795972] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:41 2019][2937743.802626] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:41 2019][2937743.808906] osp-pre-12-1 S ffff8b87966f2080 0 2391 2 0x00000080 [Fri Apr 12 11:13:41 2019][2937743.816194] Call Trace: [Fri Apr 12 11:13:41 2019][2937743.818835] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:41 2019][2937743.824245] [] schedule+0x29/0x70 [Fri Apr 12 11:13:41 2019][2937743.829414] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:41 2019][2937743.836406] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:41 2019][2937743.842208] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:41 2019][2937743.848916] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:41 2019][2937743.853976] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937743.860251] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937743.866875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937743.873146] osp-syn-13-1 S ffff8b840dfce180 0 2392 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937743.880433] Call Trace: [Fri Apr 12 11:13:42 2019][2937743.883074] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:42 2019][2937743.889212] [] schedule+0x29/0x70 [Fri Apr 12 11:13:42 2019][2937743.894384] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:42 2019][2937743.901726] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:42 2019][2937743.907498] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937743.914813] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:42 2019][2937743.920505] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:42 2019][2937743.927063] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:42 2019][2937743.934331] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:42 2019][2937743.941583] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937743.948934] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:42 2019][2937743.956722] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:42 2019][2937743.963983] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:42 2019][2937743.971753] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937743.979462] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:42 2019][2937743.985935] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:42 2019][2937743.991960] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:42 2019][2937743.998517] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.005313] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:42 2019][2937744.011756] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:42 2019][2937744.017428] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:42 2019][2937744.024998] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:42 2019][2937744.030060] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.036337] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937744.042955] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.049230] osp-pre-13-1 S ffff8b840dfcd140 0 2393 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937744.056506] Call Trace: [Fri Apr 12 11:13:42 2019][2937744.059140] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:42 2019][2937744.064545] [] schedule+0x29/0x70 [Fri Apr 12 11:13:42 2019][2937744.069700] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:42 2019][2937744.076664] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:42 2019][2937744.082421] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:42 2019][2937744.089123] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:42 2019][2937744.094236] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.100517] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937744.107138] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.113410] osp-syn-39-1 S ffff8b840dfcc100 0 2394 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937744.120694] Call Trace: [Fri Apr 12 11:13:42 2019][2937744.123325] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:42 2019][2937744.129438] [] schedule+0x29/0x70 [Fri Apr 12 11:13:42 2019][2937744.134592] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:42 2019][2937744.141914] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:42 2019][2937744.147693] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.155017] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:42 2019][2937744.160705] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:42 2019][2937744.167305] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.174559] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.181800] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.189131] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.196899] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.204146] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.211913] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.219572] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:42 2019][2937744.226016] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:42 2019][2937744.232040] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:42 2019][2937744.238616] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.245429] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:42 2019][2937744.251882] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:42 2019][2937744.257553] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:42 2019][2937744.265124] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:42 2019][2937744.270183] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.276458] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937744.283078] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.289351] osp-pre-39-1 S ffff8b840dfcb0c0 0 2395 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937744.296622] Call Trace: [Fri Apr 12 11:13:42 2019][2937744.299253] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:42 2019][2937744.304660] [] schedule+0x29/0x70 [Fri Apr 12 11:13:42 2019][2937744.309813] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:42 2019][2937744.316779] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:42 2019][2937744.322535] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:42 2019][2937744.329241] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:42 2019][2937744.334302] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.340575] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937744.342969] LustreError: dumping log to /tmp/lustre-log.1555092822.118406 [Fri Apr 12 11:13:42 2019][2937744.354156] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.360423] osp-syn-46-1 S ffff8b840dfca080 0 2396 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937744.367698] Call Trace: [Fri Apr 12 11:13:42 2019][2937744.370331] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:42 2019][2937744.376430] [] schedule+0x29/0x70 [Fri Apr 12 11:13:42 2019][2937744.381585] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:42 2019][2937744.388895] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:42 2019][2937744.394671] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.401982] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:42 2019][2937744.407655] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:42 2019][2937744.414205] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.421449] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.428696] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.436014] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:42 2019][2937744.443005] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.450769] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.458016] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.465781] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.473455] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:42 2019][2937744.479917] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:42 2019][2937744.485978] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:42 2019][2937744.492532] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.499335] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:42 2019][2937744.505786] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:42 2019][2937744.511457] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:42 2019][2937744.519073] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:42 2019][2937744.524131] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.530409] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937744.537026] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.543315] osp-pre-46-1 S ffff8b840dfc8000 0 2397 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937744.550589] Call Trace: [Fri Apr 12 11:13:42 2019][2937744.553221] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:42 2019][2937744.558625] [] schedule+0x29/0x70 [Fri Apr 12 11:13:42 2019][2937744.563784] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:42 2019][2937744.570756] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:42 2019][2937744.576513] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:42 2019][2937744.583226] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:42 2019][2937744.588288] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.594564] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937744.601200] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.607489] osp-syn-10-1 S ffff8b86c6f68000 0 2398 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937744.614784] Call Trace: [Fri Apr 12 11:13:42 2019][2937744.617416] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:42 2019][2937744.623517] [] schedule+0x29/0x70 [Fri Apr 12 11:13:42 2019][2937744.628671] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:42 2019][2937744.635991] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:42 2019][2937744.641767] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.649088] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:42 2019][2937744.654758] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:42 2019][2937744.661333] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.668576] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.675823] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.683164] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.690929] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.698174] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.705942] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.713602] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:42 2019][2937744.720046] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:42 2019][2937744.726069] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:42 2019][2937744.732648] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:42 2019][2937744.739449] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:42 2019][2937744.745892] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:42 2019][2937744.751565] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:42 2019][2937744.759138] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:42 2019][2937744.764198] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.770471] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937744.777095] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.783366] osp-pre-10-1 S ffff8b86c6f69040 0 2399 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937744.790643] Call Trace: [Fri Apr 12 11:13:42 2019][2937744.793278] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:42 2019][2937744.798684] [] schedule+0x29/0x70 [Fri Apr 12 11:13:42 2019][2937744.803842] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:42 2019][2937744.810812] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:42 2019][2937744.816577] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:42 2019][2937744.823290] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:42 2019][2937744.828350] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.834625] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:42 2019][2937744.841248] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:42 2019][2937744.847520] osp-syn-11-1 S ffff8b86c6f6a080 0 2400 2 0x00000080 [Fri Apr 12 11:13:42 2019][2937744.854841] Call Trace: [Fri Apr 12 11:13:42 2019][2937744.857481] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:43 2019][2937744.863579] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937744.868734] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:43 2019][2937744.876064] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937744.881848] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.889166] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:43 2019][2937744.894839] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937744.901398] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.908660] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.915926] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.923278] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.931042] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.938289] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.946065] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.953738] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:43 2019][2937744.960190] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:43 2019][2937744.966209] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937744.972772] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:43 2019][2937744.979585] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:43 2019][2937744.986043] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:43 2019][2937744.991714] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:43 2019][2937744.999289] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:43 2019][2937745.004383] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.010660] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:43 2019][2937745.017280] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.023559] osp-pre-11-1 S ffff8b86c6f6b0c0 0 2401 2 0x00000080 [Fri Apr 12 11:13:43 2019][2937745.030837] Call Trace: [Fri Apr 12 11:13:43 2019][2937745.033470] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:43 2019][2937745.038874] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937745.044063] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:43 2019][2937745.051038] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937745.056794] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:43 2019][2937745.063496] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:43 2019][2937745.068562] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.074835] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:43 2019][2937745.081459] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.087747] osp-syn-14-1 S ffff8b86c6f6c100 0 2402 2 0x00000080 [Fri Apr 12 11:13:43 2019][2937745.095033] Call Trace: [Fri Apr 12 11:13:43 2019][2937745.097688] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:43 2019][2937745.103795] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937745.108947] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:43 2019][2937745.116278] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937745.122060] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.129379] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:43 2019][2937745.135048] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937745.141608] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.148850] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.156096] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.163431] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.171194] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.178461] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.186231] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.193895] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:43 2019][2937745.200341] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:43 2019][2937745.206358] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937745.212907] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.219710] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:43 2019][2937745.226199] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:43 2019][2937745.231879] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:43 2019][2937745.239453] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:43 2019][2937745.244515] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.250790] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:43 2019][2937745.257412] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.263686] osp-pre-14-1 S ffff8b86c6f6d140 0 2403 2 0x00000080 [Fri Apr 12 11:13:43 2019][2937745.270972] Call Trace: [Fri Apr 12 11:13:43 2019][2937745.273612] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:43 2019][2937745.279027] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937745.284182] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:43 2019][2937745.291157] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937745.297131] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:43 2019][2937745.303847] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:43 2019][2937745.308910] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.315186] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:43 2019][2937745.321807] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.328079] osp-syn-26-1 S ffff8b86c6f6e180 0 2404 2 0x00000080 [Fri Apr 12 11:13:43 2019][2937745.335355] Call Trace: [Fri Apr 12 11:13:43 2019][2937745.337989] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:43 2019][2937745.344087] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937745.349243] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:43 2019][2937745.356563] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937745.362358] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.369676] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:43 2019][2937745.375345] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937745.381896] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.389151] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.396394] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.403727] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.411492] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.418784] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.426548] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.434209] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:43 2019][2937745.440657] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:43 2019][2937745.446676] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937745.453235] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.460032] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:43 2019][2937745.466676] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:43 2019][2937745.472371] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:43 2019][2937745.479952] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:43 2019][2937745.485022] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.491334] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:43 2019][2937745.497953] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.504236] osp-pre-26-1 S ffff8b8b75f78000 0 2405 2 0x00000080 [Fri Apr 12 11:13:43 2019][2937745.511514] Call Trace: [Fri Apr 12 11:13:43 2019][2937745.514153] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:43 2019][2937745.519572] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937745.524740] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:43 2019][2937745.531711] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937745.537487] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:43 2019][2937745.544207] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:43 2019][2937745.549285] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.555562] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:43 2019][2937745.562183] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.568458] osp-syn-31-1 S ffff8b8b75f79040 0 2406 2 0x00000080 [Fri Apr 12 11:13:43 2019][2937745.575749] Call Trace: [Fri Apr 12 11:13:43 2019][2937745.578383] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:43 2019][2937745.584484] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937745.589639] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:43 2019][2937745.596956] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937745.602730] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.610077] [] ? llog_open+0x140/0x3d0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.616634] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937745.623200] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.630452] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.637713] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.645063] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.652830] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.660501] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.668284] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.676152] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:43 2019][2937745.683011] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:43 2019][2937745.689038] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937745.695599] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.702422] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:43 2019][2937745.708878] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:43 2019][2937745.714553] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:43 2019][2937745.722130] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:43 2019][2937745.727815] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.734269] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:43 2019][2937745.740908] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.747184] osp-pre-31-1 S ffff8b8b75f7a080 0 2407 2 0x00000080 [Fri Apr 12 11:13:43 2019][2937745.754459] Call Trace: [Fri Apr 12 11:13:43 2019][2937745.757100] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:43 2019][2937745.762506] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937745.767662] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:43 2019][2937745.774634] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937745.780389] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:43 2019][2937745.787093] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:43 2019][2937745.792154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.798430] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:43 2019][2937745.805070] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:43 2019][2937745.811341] osp-syn-42-1 S ffff8b8b75f7b0c0 0 2408 2 0x00000080 [Fri Apr 12 11:13:43 2019][2937745.818611] Call Trace: [Fri Apr 12 11:13:43 2019][2937745.821240] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:43 2019][2937745.827342] [] schedule+0x29/0x70 [Fri Apr 12 11:13:43 2019][2937745.832500] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:43 2019][2937745.839817] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:43 2019][2937745.845592] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:43 2019][2937745.852902] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:43 2019][2937745.858574] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:43 2019][2937745.865139] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937745.872385] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:44 2019][2937745.879631] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937745.886966] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937745.897107] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937745.904548] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937745.912317] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937745.919979] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:44 2019][2937745.926421] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:44 2019][2937745.932441] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:44 2019][2937745.938993] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:44 2019][2937745.945787] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:44 2019][2937745.952237] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:44 2019][2937745.957908] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:44 2019][2937745.965497] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:44 2019][2937745.970564] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937745.976840] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:44 2019][2937745.983458] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937745.989731] osp-pre-42-1 S ffff8b8b75f7c100 0 2409 2 0x00000080 [Fri Apr 12 11:13:44 2019][2937745.997002] Call Trace: [Fri Apr 12 11:13:44 2019][2937745.999633] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:44 2019][2937746.005039] [] schedule+0x29/0x70 [Fri Apr 12 11:13:44 2019][2937746.010193] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:44 2019][2937746.017159] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:44 2019][2937746.022954] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:44 2019][2937746.029693] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:44 2019][2937746.034760] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.041044] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:44 2019][2937746.047673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.053946] osp-syn-9-1 S ffff8b8b75f7d140 0 2410 2 0x00000080 [Fri Apr 12 11:13:44 2019][2937746.061248] Call Trace: [Fri Apr 12 11:13:44 2019][2937746.063892] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:44 2019][2937746.069991] [] schedule+0x29/0x70 [Fri Apr 12 11:13:44 2019][2937746.075156] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:44 2019][2937746.082482] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:44 2019][2937746.088257] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.095582] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:44 2019][2937746.102137] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.109383] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.116628] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.123959] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.131725] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.141012] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.149027] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.156710] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:44 2019][2937746.163162] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:44 2019][2937746.169181] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:44 2019][2937746.175740] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.182571] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:44 2019][2937746.189024] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:44 2019][2937746.194695] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:44 2019][2937746.202279] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:44 2019][2937746.207342] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.213620] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:44 2019][2937746.220240] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.226514] osp-pre-9-1 S ffff8b8b75f7e180 0 2411 2 0x00000080 [Fri Apr 12 11:13:44 2019][2937746.233789] Call Trace: [Fri Apr 12 11:13:44 2019][2937746.236423] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:44 2019][2937746.241829] [] schedule+0x29/0x70 [Fri Apr 12 11:13:44 2019][2937746.246988] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:44 2019][2937746.253956] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:44 2019][2937746.259714] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:44 2019][2937746.266427] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:44 2019][2937746.271493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.277774] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:44 2019][2937746.284409] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.290691] osp-syn-2-1 S ffff8b853bb70000 0 2412 2 0x00000080 [Fri Apr 12 11:13:44 2019][2937746.297992] Call Trace: [Fri Apr 12 11:13:44 2019][2937746.300627] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:44 2019][2937746.306728] [] schedule+0x29/0x70 [Fri Apr 12 11:13:44 2019][2937746.311916] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:44 2019][2937746.319953] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:44 2019][2937746.325810] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.333143] [] ? llog_open+0x140/0x3d0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.339683] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:44 2019][2937746.346239] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.353484] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.360739] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.368073] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.375853] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.383129] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.390902] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.398563] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:44 2019][2937746.405021] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:44 2019][2937746.411054] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:44 2019][2937746.417613] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.424412] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:44 2019][2937746.430874] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:44 2019][2937746.436543] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:44 2019][2937746.444194] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:44 2019][2937746.449282] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.455567] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:44 2019][2937746.462206] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.468511] osp-pre-2-1 S ffff8b853bb71040 0 2413 2 0x00000080 [Fri Apr 12 11:13:44 2019][2937746.475794] Call Trace: [Fri Apr 12 11:13:44 2019][2937746.478429] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:44 2019][2937746.483834] [] schedule+0x29/0x70 [Fri Apr 12 11:13:44 2019][2937746.488997] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:44 2019][2937746.495966] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:44 2019][2937746.501726] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:44 2019][2937746.508429] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:44 2019][2937746.513491] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.519778] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:44 2019][2937746.526396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.532670] osp-syn-43-1 S ffff8b853bb72080 0 2414 2 0x00000080 [Fri Apr 12 11:13:44 2019][2937746.539956] Call Trace: [Fri Apr 12 11:13:44 2019][2937746.542586] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:44 2019][2937746.548706] [] schedule+0x29/0x70 [Fri Apr 12 11:13:44 2019][2937746.553870] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:44 2019][2937746.561187] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:44 2019][2937746.566987] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.574324] [] ? llog_open+0x140/0x3d0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.580869] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:44 2019][2937746.587431] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.594990] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.602239] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.609583] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.617356] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.624601] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.632366] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.640057] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:44 2019][2937746.646504] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:44 2019][2937746.652531] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:44 2019][2937746.659096] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.665922] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:44 2019][2937746.672391] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:44 2019][2937746.678064] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:44 2019][2937746.685662] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:44 2019][2937746.690719] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.696999] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:44 2019][2937746.703637] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.709908] osp-pre-43-1 S ffff8b853bb730c0 0 2415 2 0x00000080 [Fri Apr 12 11:13:44 2019][2937746.717192] Call Trace: [Fri Apr 12 11:13:44 2019][2937746.719822] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:44 2019][2937746.725232] [] schedule+0x29/0x70 [Fri Apr 12 11:13:44 2019][2937746.730452] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:44 2019][2937746.737420] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:44 2019][2937746.743178] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:44 2019][2937746.749890] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:44 2019][2937746.754952] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.761253] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:44 2019][2937746.767873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:44 2019][2937746.774147] osp-syn-19-1 S ffff8b853bb74100 0 2416 2 0x00000080 [Fri Apr 12 11:13:44 2019][2937746.781431] Call Trace: [Fri Apr 12 11:13:44 2019][2937746.784078] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:44 2019][2937746.790227] [] schedule+0x29/0x70 [Fri Apr 12 11:13:44 2019][2937746.795402] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:44 2019][2937746.802727] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:44 2019][2937746.808510] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.815836] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:44 2019][2937746.821515] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:44 2019][2937746.828075] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.835326] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.842574] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.849895] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:44 2019][2937746.856884] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.864661] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:44 2019][2937746.871909] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937746.879681] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937746.887350] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:45 2019][2937746.893827] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937746.900384] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:45 2019][2937746.907185] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:45 2019][2937746.913636] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:45 2019][2937746.919307] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:45 2019][2937746.926884] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:45 2019][2937746.931944] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937746.938220] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:45 2019][2937746.944841] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937746.951115] osp-pre-19-1 S ffff8b853bb75140 0 2417 2 0x00000080 [Fri Apr 12 11:13:45 2019][2937746.958407] Call Trace: [Fri Apr 12 11:13:45 2019][2937746.961042] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:45 2019][2937746.966449] [] schedule+0x29/0x70 [Fri Apr 12 11:13:45 2019][2937746.971601] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:45 2019][2937746.978576] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:45 2019][2937746.984333] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:45 2019][2937746.991045] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:45 2019][2937746.996103] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.002380] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:45 2019][2937747.009002] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.015276] osp-syn-18-1 S ffff8b853bb76180 0 2418 2 0x00000080 [Fri Apr 12 11:13:45 2019][2937747.022550] Call Trace: [Fri Apr 12 11:13:45 2019][2937747.025185] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:45 2019][2937747.031286] [] schedule+0x29/0x70 [Fri Apr 12 11:13:45 2019][2937747.036437] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:45 2019][2937747.043758] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:45 2019][2937747.049534] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.056860] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937747.063417] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.070662] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.077909] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.085252] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.093032] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.100287] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.108061] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.115729] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:45 2019][2937747.122174] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:45 2019][2937747.128196] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937747.134756] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.141563] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:45 2019][2937747.148013] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:45 2019][2937747.153686] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:45 2019][2937747.161262] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:45 2019][2937747.166323] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.172598] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:45 2019][2937747.179220] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.185493] osp-pre-18-1 S ffff8b8537300000 0 2419 2 0x00000080 [Fri Apr 12 11:13:45 2019][2937747.192768] Call Trace: [Fri Apr 12 11:13:45 2019][2937747.195401] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:45 2019][2937747.200810] [] schedule+0x29/0x70 [Fri Apr 12 11:13:45 2019][2937747.205991] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:45 2019][2937747.212963] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:45 2019][2937747.218719] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:45 2019][2937747.225433] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:45 2019][2937747.230493] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.236770] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:45 2019][2937747.243388] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.249664] osp-syn-25-1 S ffff8b8537301040 0 2420 2 0x00000080 [Fri Apr 12 11:13:45 2019][2937747.256938] Call Trace: [Fri Apr 12 11:13:45 2019][2937747.259570] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:45 2019][2937747.265674] [] schedule+0x29/0x70 [Fri Apr 12 11:13:45 2019][2937747.270825] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:45 2019][2937747.278146] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:45 2019][2937747.283919] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.291240] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937747.297794] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.305047] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.312295] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.319635] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.327402] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.334656] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.342430] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.350098] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:45 2019][2937747.356588] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:45 2019][2937747.362609] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937747.369167] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.375975] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:45 2019][2937747.382423] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:45 2019][2937747.388098] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:45 2019][2937747.395673] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:45 2019][2937747.400736] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.407011] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:45 2019][2937747.413630] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.419905] osp-pre-25-1 S ffff8b8537302080 0 2421 2 0x00000080 [Fri Apr 12 11:13:45 2019][2937747.427189] Call Trace: [Fri Apr 12 11:13:45 2019][2937747.429823] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:45 2019][2937747.435232] [] schedule+0x29/0x70 [Fri Apr 12 11:13:45 2019][2937747.440385] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:45 2019][2937747.447357] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:45 2019][2937747.453115] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:45 2019][2937747.459828] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:45 2019][2937747.464887] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.471165] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:45 2019][2937747.477783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.484058] osp-syn-17-1 S ffff8b85373030c0 0 2422 2 0x00000080 [Fri Apr 12 11:13:45 2019][2937747.491332] Call Trace: [Fri Apr 12 11:13:45 2019][2937747.493964] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:45 2019][2937747.500067] [] schedule+0x29/0x70 [Fri Apr 12 11:13:45 2019][2937747.505221] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:45 2019][2937747.512540] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:45 2019][2937747.518317] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.525641] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937747.532197] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.539444] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.546689] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.554065] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.561823] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.569079] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.576848] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.584511] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:45 2019][2937747.590956] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:45 2019][2937747.596978] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937747.603539] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.610340] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:45 2019][2937747.616795] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:45 2019][2937747.622464] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:45 2019][2937747.630038] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:45 2019][2937747.635096] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.641370] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:45 2019][2937747.647992] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.654266] osp-pre-17-1 S ffff8b8537304100 0 2423 2 0x00000080 [Fri Apr 12 11:13:45 2019][2937747.661541] Call Trace: [Fri Apr 12 11:13:45 2019][2937747.664173] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:45 2019][2937747.669581] [] schedule+0x29/0x70 [Fri Apr 12 11:13:45 2019][2937747.674734] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:45 2019][2937747.681702] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:45 2019][2937747.687460] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:45 2019][2937747.694171] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:45 2019][2937747.699231] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.705509] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:45 2019][2937747.712152] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:45 2019][2937747.718427] osp-syn-0-1 S ffff8b8537305140 0 2424 2 0x00000080 [Fri Apr 12 11:13:45 2019][2937747.725704] Call Trace: [Fri Apr 12 11:13:45 2019][2937747.728334] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:45 2019][2937747.734438] [] schedule+0x29/0x70 [Fri Apr 12 11:13:45 2019][2937747.739590] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:45 2019][2937747.746910] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:45 2019][2937747.752688] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.760013] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937747.766563] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.773816] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.781066] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.788388] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:45 2019][2937747.795376] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.803150] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.810425] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.818202] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.825884] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:45 2019][2937747.832349] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:45 2019][2937747.838369] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:45 2019][2937747.844979] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:45 2019][2937747.851820] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:45 2019][2937747.858272] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:45 2019][2937747.863939] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:45 2019][2937747.871512] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937747.876569] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937747.882858] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:46 2019][2937747.889489] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937747.895758] osp-pre-0-1 S ffff8b8537306180 0 2425 2 0x00000080 [Fri Apr 12 11:13:46 2019][2937747.903035] Call Trace: [Fri Apr 12 11:13:46 2019][2937747.905670] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:46 2019][2937747.911078] [] schedule+0x29/0x70 [Fri Apr 12 11:13:46 2019][2937747.916230] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:46 2019][2937747.923215] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:46 2019][2937747.928978] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:46 2019][2937747.935689] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937747.940750] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937747.947028] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:46 2019][2937747.953646] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937747.959917] osp-syn-5-1 S ffff8b84e42d8000 0 2426 2 0x00000080 [Fri Apr 12 11:13:46 2019][2937747.967205] Call Trace: [Fri Apr 12 11:13:46 2019][2937747.969837] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:46 2019][2937747.975935] [] schedule+0x29/0x70 [Fri Apr 12 11:13:46 2019][2937747.981090] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:46 2019][2937747.988401] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:46 2019][2937747.994177] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.001540] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:46 2019][2937748.009766] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.017035] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.024280] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.031632] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.039399] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.046678] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.054443] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.062108] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:46 2019][2937748.068579] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:46 2019][2937748.074624] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:46 2019][2937748.081184] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.087991] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:46 2019][2937748.094443] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:46 2019][2937748.100130] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:46 2019][2937748.107700] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937748.112761] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.119037] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:46 2019][2937748.125656] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.131930] osp-pre-5-1 S ffff8b84e42d9040 0 2427 2 0x00000080 [Fri Apr 12 11:13:46 2019][2937748.139199] Call Trace: [Fri Apr 12 11:13:46 2019][2937748.141831] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:46 2019][2937748.147236] [] schedule+0x29/0x70 [Fri Apr 12 11:13:46 2019][2937748.152389] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:46 2019][2937748.159374] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:46 2019][2937748.165130] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:46 2019][2937748.171833] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937748.176896] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.183170] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:46 2019][2937748.189791] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.196064] osp-syn-44-1 S ffff8b84e42da080 0 2428 2 0x00000080 [Fri Apr 12 11:13:46 2019][2937748.203335] Call Trace: [Fri Apr 12 11:13:46 2019][2937748.205965] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:46 2019][2937748.212067] [] schedule+0x29/0x70 [Fri Apr 12 11:13:46 2019][2937748.217219] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:46 2019][2937748.224528] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:46 2019][2937748.230303] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.237615] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:46 2019][2937748.243289] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:46 2019][2937748.249837] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.257083] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.264328] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.271671] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.279436] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.286684] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.294446] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.302109] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:46 2019][2937748.308565] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:46 2019][2937748.314605] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:46 2019][2937748.321162] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.327975] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:46 2019][2937748.334436] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:46 2019][2937748.340108] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:46 2019][2937748.347697] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937748.352759] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.359040] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:46 2019][2937748.365660] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.371934] osp-pre-44-1 S ffff8b84e42db0c0 0 2429 2 0x00000080 [Fri Apr 12 11:13:46 2019][2937748.379228] Call Trace: [Fri Apr 12 11:13:46 2019][2937748.381861] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:46 2019][2937748.387268] [] schedule+0x29/0x70 [Fri Apr 12 11:13:46 2019][2937748.392429] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:46 2019][2937748.399403] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:46 2019][2937748.405161] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:46 2019][2937748.411893] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937748.416961] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.423238] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:46 2019][2937748.429869] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.436147] osp-syn-32-1 S[2937748.439036] LustreError: dumping log to /tmp/lustre-log.1555092826.118312 [Fri Apr 12 11:13:46 2019] ffff8b84e42dc100 0 2430 2 0x00000080 [Fri Apr 12 11:13:46 2019][2937748.450409] Call Trace: [Fri Apr 12 11:13:46 2019][2937748.453053] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:46 2019][2937748.459194] [] schedule+0x29/0x70 [Fri Apr 12 11:13:46 2019][2937748.464349] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:46 2019][2937748.471702] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:46 2019][2937748.477482] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.484807] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:46 2019][2937748.490480] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:46 2019][2937748.497040] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.504288] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.511538] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.518872] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:46 2019][2937748.525893] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.533662] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.540912] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.548680] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.556342] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:46 2019][2937748.562800] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:46 2019][2937748.568828] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:46 2019][2937748.575395] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.582207] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:46 2019][2937748.588689] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:46 2019][2937748.594359] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:46 2019][2937748.601940] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937748.607000] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.613293] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:46 2019][2937748.619934] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.626211] osp-pre-32-1 S ffff8b84e42dd140 0 2431 2 0x00000080 [Fri Apr 12 11:13:46 2019][2937748.633486] Call Trace: [Fri Apr 12 11:13:46 2019][2937748.636119] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:46 2019][2937748.641528] [] schedule+0x29/0x70 [Fri Apr 12 11:13:46 2019][2937748.647473] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:46 2019][2937748.654842] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:46 2019][2937748.660602] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:46 2019][2937748.667312] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937748.672382] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.678657] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:46 2019][2937748.685277] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.691560] osp-syn-21-1 S ffff8b84e42de180 0 2432 2 0x00000080 [Fri Apr 12 11:13:46 2019][2937748.698855] Call Trace: [Fri Apr 12 11:13:46 2019][2937748.701495] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:46 2019][2937748.707604] [] schedule+0x29/0x70 [Fri Apr 12 11:13:46 2019][2937748.712797] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:46 2019][2937748.720114] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:46 2019][2937748.725906] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.733223] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:46 2019][2937748.738893] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:46 2019][2937748.745444] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.752690] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.759936] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.767268] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.775037] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.782287] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.790056] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.797713] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:46 2019][2937748.804160] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:46 2019][2937748.810178] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:46 2019][2937748.816729] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:46 2019][2937748.823528] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:46 2019][2937748.829981] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:46 2019][2937748.835657] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:46 2019][2937748.843247] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:46 2019][2937748.848320] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:46 2019][2937748.854595] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937748.861216] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937748.867498] osp-pre-21-1 S ffff8b8bc1630000 0 2433 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937748.874780] Call Trace: [Fri Apr 12 11:13:47 2019][2937748.877418] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:47 2019][2937748.882824] [] schedule+0x29/0x70 [Fri Apr 12 11:13:47 2019][2937748.887978] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:47 2019][2937748.894956] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:47 2019][2937748.900716] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:47 2019][2937748.907439] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:47 2019][2937748.912496] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937748.918792] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937748.925411] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937748.931685] osp-syn-16-1 S ffff8b8bc1631040 0 2434 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937748.938977] Call Trace: [Fri Apr 12 11:13:47 2019][2937748.941611] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:47 2019][2937748.947712] [] schedule+0x29/0x70 [Fri Apr 12 11:13:47 2019][2937748.952875] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:47 2019][2937748.960204] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:47 2019][2937748.966004] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937748.973323] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:47 2019][2937748.979002] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:47 2019][2937748.985625] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:47 2019][2937748.992879] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.000129] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.007450] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:47 2019][2937749.014459] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.022238] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.029484] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.037250] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.044915] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:47 2019][2937749.051386] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:47 2019][2937749.057404] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:47 2019][2937749.063973] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.070786] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:47 2019][2937749.077239] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:47 2019][2937749.082909] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:47 2019][2937749.090481] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:47 2019][2937749.095542] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.102216] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937749.108996] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.115275] osp-pre-16-1 S ffff8b8bc1632080 0 2435 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937749.122785] Call Trace: [Fri Apr 12 11:13:47 2019][2937749.125422] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:47 2019][2937749.130835] [] schedule+0x29/0x70 [Fri Apr 12 11:13:47 2019][2937749.135991] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:47 2019][2937749.142961] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:47 2019][2937749.148715] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:47 2019][2937749.155419] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:47 2019][2937749.160479] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.166756] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937749.173378] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.179650] osp-syn-41-1 S ffff8b8bc16330c0 0 2436 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937749.186927] Call Trace: [Fri Apr 12 11:13:47 2019][2937749.189559] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:47 2019][2937749.195660] [] schedule+0x29/0x70 [Fri Apr 12 11:13:47 2019][2937749.200811] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:47 2019][2937749.208127] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:47 2019][2937749.213925] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.221248] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:47 2019][2937749.226920] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:47 2019][2937749.233481] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.240741] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.247992] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.255314] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:47 2019][2937749.262305] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.270084] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.277329] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.285097] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.292767] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:47 2019][2937749.299214] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:47 2019][2937749.305233] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:47 2019][2937749.311802] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.318629] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:47 2019][2937749.325087] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:47 2019][2937749.330775] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:47 2019][2937749.338363] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:47 2019][2937749.343421] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.349695] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937749.356319] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.362594] osp-pre-41-1 S ffff8b8bc1634100 0 2437 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937749.369904] Call Trace: [Fri Apr 12 11:13:47 2019][2937749.372537] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:47 2019][2937749.377964] [] schedule+0x29/0x70 [Fri Apr 12 11:13:47 2019][2937749.383124] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:47 2019][2937749.390094] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:47 2019][2937749.395856] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:47 2019][2937749.402567] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:47 2019][2937749.407635] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.413911] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937749.420559] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.426851] osp-syn-4-1 S ffff8b8bc1635140 0 2438 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937749.434136] Call Trace: [Fri Apr 12 11:13:47 2019][2937749.436773] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:47 2019][2937749.442876] [] schedule+0x29/0x70 [Fri Apr 12 11:13:47 2019][2937749.448041] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:47 2019][2937749.455400] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:47 2019][2937749.461182] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.468505] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:47 2019][2937749.474176] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:47 2019][2937749.480737] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.488011] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.495257] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.502591] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.510364] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.517625] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.525398] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.533064] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:47 2019][2937749.539512] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:47 2019][2937749.545533] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:47 2019][2937749.552091] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.558899] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:47 2019][2937749.565349] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:47 2019][2937749.571019] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:47 2019][2937749.578600] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:47 2019][2937749.584207] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.590653] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937749.597278] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.603555] osp-pre-4-1 S ffff8b8bc1636180 0 2439 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937749.610832] Call Trace: [Fri Apr 12 11:13:47 2019][2937749.613466] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:47 2019][2937749.618876] [] schedule+0x29/0x70 [Fri Apr 12 11:13:47 2019][2937749.624026] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:47 2019][2937749.630998] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:47 2019][2937749.636757] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:47 2019][2937749.643487] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:47 2019][2937749.648545] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.654823] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937749.661441] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.667715] osp-syn-23-1 S ffff8b8479a98000 0 2440 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937749.674985] Call Trace: [Fri Apr 12 11:13:47 2019][2937749.677617] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:47 2019][2937749.683717] [] schedule+0x29/0x70 [Fri Apr 12 11:13:47 2019][2937749.688871] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:47 2019][2937749.696179] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:47 2019][2937749.701955] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.709269] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:47 2019][2937749.714939] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:47 2019][2937749.721489] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.728733] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.735978] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.743312] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.751081] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.758325] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.766086] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.773747] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:47 2019][2937749.780194] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:47 2019][2937749.786213] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:47 2019][2937749.792764] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:47 2019][2937749.799563] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:47 2019][2937749.806007] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:47 2019][2937749.811677] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:47 2019][2937749.819280] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:47 2019][2937749.824411] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.830685] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:47 2019][2937749.837334] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:47 2019][2937749.843614] osp-pre-23-1 S ffff8b8479a99040 0 2441 2 0x00000080 [Fri Apr 12 11:13:47 2019][2937749.850888] Call Trace: [Fri Apr 12 11:13:47 2019][2937749.853521] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:47 2019][2937749.858926] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937749.864083] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:48 2019][2937749.871079] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937749.876858] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:48 2019][2937749.883577] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937749.888640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937749.894916] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:48 2019][2937749.901551] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937749.907833] osp-syn-33-1 S ffff8b8479a9a080 0 2442 2 0x00000080 [Fri Apr 12 11:13:48 2019][2937749.915131] Call Trace: [Fri Apr 12 11:13:48 2019][2937749.917767] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:48 2019][2937749.923877] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937749.929056] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:48 2019][2937749.936377] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937749.942167] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937749.949497] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:48 2019][2937749.956066] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:48 2019][2937749.963345] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:48 2019][2937749.970610] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937749.977944] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:48 2019][2937749.984940] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:48 2019][2937749.992710] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.000078] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.007867] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.015551] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:48 2019][2937750.022013] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:48 2019][2937750.028035] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:48 2019][2937750.034601] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.041394] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:48 2019][2937750.047843] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:48 2019][2937750.053526] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:48 2019][2937750.061170] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937750.066249] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.072527] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:48 2019][2937750.079159] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.085471] osp-pre-33-1 S ffff8b8479a9b0c0 0 2443 2 0x00000080 [Fri Apr 12 11:13:48 2019][2937750.092754] Call Trace: [Fri Apr 12 11:13:48 2019][2937750.095388] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:48 2019][2937750.100795] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937750.105968] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:48 2019][2937750.112938] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937750.118701] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:48 2019][2937750.125408] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937750.130488] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.136771] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:48 2019][2937750.143399] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.149673] osp-syn-20-1 S ffff8b8479a9c100 0 2444 2 0x00000080 [Fri Apr 12 11:13:48 2019][2937750.156949] Call Trace: [Fri Apr 12 11:13:48 2019][2937750.159580] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:48 2019][2937750.165682] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937750.170861] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:48 2019][2937750.178182] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937750.183962] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.191287] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:48 2019][2937750.196956] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:48 2019][2937750.203507] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.210755] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.218022] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.225355] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.233129] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.240373] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.248141] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.255803] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:48 2019][2937750.262247] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:48 2019][2937750.268269] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:48 2019][2937750.274829] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.281625] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:48 2019][2937750.288080] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:48 2019][2937750.293760] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:48 2019][2937750.301335] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937750.306396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.312671] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:48 2019][2937750.319332] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.325603] osp-pre-20-1 S ffff8b8479a9d140 0 2445 2 0x00000080 [Fri Apr 12 11:13:48 2019][2937750.332908] Call Trace: [Fri Apr 12 11:13:48 2019][2937750.335557] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:48 2019][2937750.340970] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937750.346125] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:48 2019][2937750.353097] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937750.358853] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:48 2019][2937750.365596] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937750.370664] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.376939] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:48 2019][2937750.383576] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.389868] osp-syn-7-1 S ffff8b8479a9e180 0 2446 2 0x00000080 [Fri Apr 12 11:13:48 2019][2937750.397143] Call Trace: [Fri Apr 12 11:13:48 2019][2937750.399775] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:48 2019][2937750.405877] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937750.411039] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:48 2019][2937750.418379] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937750.424170] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.431492] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:48 2019][2937750.438071] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.445329] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.452594] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.459925] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.467697] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.474944] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.482752] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.487100] LustreError: dumping log to /tmp/lustre-log.1555092828.118516 [Fri Apr 12 11:13:48 2019][2937750.497407] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:48 2019][2937750.503854] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:48 2019][2937750.509895] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:48 2019][2937750.516461] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.523266] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:48 2019][2937750.529719] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:48 2019][2937750.535391] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:48 2019][2937750.543138] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937750.548203] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.554480] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:48 2019][2937750.561101] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.567422] osp-pre-7-1 S ffff8b87fa230000 0 2447 2 0x00000080 [Fri Apr 12 11:13:48 2019][2937750.574701] Call Trace: [Fri Apr 12 11:13:48 2019][2937750.577336] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:48 2019][2937750.582740] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937750.587898] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:48 2019][2937750.594864] [] ? sched_clock+0x9/0x10 [Fri Apr 12 11:13:48 2019][2937750.600360] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937750.606120] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:48 2019][2937750.612821] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937750.617886] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.624178] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:48 2019][2937750.630799] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.637074] osp-syn-40-1 S ffff8b87fa231040 0 2448 2 0x00000080 [Fri Apr 12 11:13:48 2019][2937750.644357] Call Trace: [Fri Apr 12 11:13:48 2019][2937750.646991] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:48 2019][2937750.653099] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937750.658268] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:48 2019][2937750.665609] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937750.671400] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.678740] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:48 2019][2937750.685298] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.692541] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.699789] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.707110] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:48 2019][2937750.714100] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.721873] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.729136] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.736911] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.744572] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:48 2019][2937750.751018] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:48 2019][2937750.757038] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:48 2019][2937750.763598] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:48 2019][2937750.770400] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:48 2019][2937750.776845] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:48 2019][2937750.782518] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:48 2019][2937750.790097] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937750.795154] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.801431] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:48 2019][2937750.808051] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:48 2019][2937750.814324] osp-pre-40-1 S ffff8b87fa232080 0 2449 2 0x00000080 [Fri Apr 12 11:13:48 2019][2937750.821593] Call Trace: [Fri Apr 12 11:13:48 2019][2937750.824224] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:48 2019][2937750.829631] [] schedule+0x29/0x70 [Fri Apr 12 11:13:48 2019][2937750.834787] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:48 2019][2937750.841762] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:48 2019][2937750.847517] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:48 2019][2937750.854220] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:48 2019][2937750.859295] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937750.865570] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937750.872196] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937750.878480] osp-syn-45-1 S ffff8b87fa2330c0 0 2450 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937750.885765] Call Trace: [Fri Apr 12 11:13:49 2019][2937750.888416] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:49 2019][2937750.894514] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937750.899668] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:49 2019][2937750.906986] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937750.912759] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937750.920094] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:49 2019][2937750.926670] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:49 2019][2937750.933917] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:49 2019][2937750.941188] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937750.948540] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:49 2019][2937750.956334] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:49 2019][2937750.963582] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:49 2019][2937750.971346] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937750.979009] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:49 2019][2937750.985455] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:49 2019][2937750.991475] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:49 2019][2937750.998025] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.004821] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:49 2019][2937751.011276] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:49 2019][2937751.016964] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:49 2019][2937751.024555] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:49 2019][2937751.029763] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.036057] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937751.042694] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.048973] osp-pre-45-1 S ffff8b87fa234100 0 2451 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937751.056258] Call Trace: [Fri Apr 12 11:13:49 2019][2937751.058892] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:49 2019][2937751.064298] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937751.069451] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:49 2019][2937751.076443] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937751.082198] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:49 2019][2937751.088910] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:49 2019][2937751.093974] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.100249] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937751.106898] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.113168] osp-syn-6-1 S ffff8b87fa235140 0 2452 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937751.120549] Call Trace: [Fri Apr 12 11:13:49 2019][2937751.123183] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:49 2019][2937751.129295] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937751.134445] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:49 2019][2937751.141777] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937751.147558] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.154882] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:49 2019][2937751.161448] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.168711] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.175963] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.183296] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.191061] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.198306] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.206091] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.213759] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:49 2019][2937751.220209] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:49 2019][2937751.226226] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:49 2019][2937751.232788] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.239601] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:49 2019][2937751.246061] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:49 2019][2937751.251745] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:49 2019][2937751.259322] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:49 2019][2937751.264379] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.270653] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937751.277280] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.283550] osp-pre-6-1 S ffff8b87fa236180 0 2453 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937751.290827] Call Trace: [Fri Apr 12 11:13:49 2019][2937751.293460] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:49 2019][2937751.298867] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937751.304023] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:49 2019][2937751.311028] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937751.316784] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:49 2019][2937751.323488] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:49 2019][2937751.328682] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.334966] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937751.341598] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.347868] osp-syn-24-1 S ffff8b891db30000 0 2454 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937751.355143] Call Trace: [Fri Apr 12 11:13:49 2019][2937751.357778] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:49 2019][2937751.363885] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937751.369041] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:49 2019][2937751.376364] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937751.382144] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.389490] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:49 2019][2937751.396044] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.403322] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.410578] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.417935] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.425716] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.432991] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.440764] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.448451] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:49 2019][2937751.454895] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:49 2019][2937751.460920] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:49 2019][2937751.467484] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.474296] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:49 2019][2937751.480749] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:49 2019][2937751.486466] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:49 2019][2937751.494056] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:49 2019][2937751.499114] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.505417] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937751.512037] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.518310] osp-pre-24-1 S ffff8b891db31040 0 2455 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937751.525595] Call Trace: [Fri Apr 12 11:13:49 2019][2937751.528230] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:49 2019][2937751.533634] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937751.538800] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:49 2019][2937751.545771] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937751.551526] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:49 2019][2937751.558241] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:49 2019][2937751.563319] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.569611] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937751.576245] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.582542] osp-syn-47-1 S ffff8b891db32080 0 2456 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937751.589836] Call Trace: [Fri Apr 12 11:13:49 2019][2937751.592485] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:49 2019][2937751.598604] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937751.603756] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:49 2019][2937751.611086] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937751.616876] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.624199] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:49 2019][2937751.629867] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:49 2019][2937751.636447] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.643727] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.650985] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.658335] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:49 2019][2937751.665324] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.673100] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.680346] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.688123] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.695807] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:49 2019][2937751.702284] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:49 2019][2937751.708314] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:49 2019][2937751.714872] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:49 2019][2937751.721677] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:49 2019][2937751.728128] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:49 2019][2937751.733800] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:49 2019][2937751.741403] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:49 2019][2937751.746486] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.752762] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937751.759385] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.765663] osp-pre-47-1 S ffff8b891db330c0 0 2457 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937751.772947] Call Trace: [Fri Apr 12 11:13:49 2019][2937751.775581] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:49 2019][2937751.780993] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937751.786150] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:49 2019][2937751.793132] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937751.798891] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:49 2019][2937751.805600] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:49 2019][2937751.810663] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.816940] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:49 2019][2937751.823561] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:49 2019][2937751.829831] osp-syn-38-1 S ffff8b891db34100 0 2458 2 0x00000080 [Fri Apr 12 11:13:49 2019][2937751.837133] Call Trace: [Fri Apr 12 11:13:49 2019][2937751.839765] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:49 2019][2937751.845868] [] schedule+0x29/0x70 [Fri Apr 12 11:13:49 2019][2937751.851040] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:49 2019][2937751.858374] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:49 2019][2937751.864150] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937751.871471] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:50 2019][2937751.877147] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937751.883739] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937751.890991] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:50 2019][2937751.898253] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937751.905588] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:50 2019][2937751.913379] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937751.920637] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:50 2019][2937751.928414] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937751.936075] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:50 2019][2937751.942547] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:50 2019][2937751.948607] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937751.955179] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:50 2019][2937751.968453] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:50 2019][2937751.974125] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:50 2019][2937751.981714] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:50 2019][2937751.986783] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937751.993057] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:50 2019][2937751.999676] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.005949] osp-pre-38-1 S ffff8b891db35140 0 2459 2 0x00000080 [Fri Apr 12 11:13:50 2019][2937752.013225] Call Trace: [Fri Apr 12 11:13:50 2019][2937752.015858] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:50 2019][2937752.021266] [] schedule+0x29/0x70 [Fri Apr 12 11:13:50 2019][2937752.026422] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:50 2019][2937752.033390] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:50 2019][2937752.039150] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:50 2019][2937752.045854] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:50 2019][2937752.050914] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.057190] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:50 2019][2937752.063811] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.070085] osp-syn-15-1 S ffff8b891db36180 0 2460 2 0x00000080 [Fri Apr 12 11:13:50 2019][2937752.077360] Call Trace: [Fri Apr 12 11:13:50 2019][2937752.079991] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:50 2019][2937752.086096] [] schedule+0x29/0x70 [Fri Apr 12 11:13:50 2019][2937752.091247] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:50 2019][2937752.098559] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:50 2019][2937752.104332] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.111644] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:50 2019][2937752.117315] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937752.123867] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.131114] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.138364] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.145698] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.153463] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.160710] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.168474] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.176136] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:50 2019][2937752.182580] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:50 2019][2937752.188600] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937752.195151] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.201949] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:50 2019][2937752.208393] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:50 2019][2937752.214063] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:50 2019][2937752.221635] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:50 2019][2937752.226695] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.232971] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:50 2019][2937752.239592] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.245865] osp-pre-15-1 S ffff8b8cc2638000 0 2461 2 0x00000080 [Fri Apr 12 11:13:50 2019][2937752.253142] Call Trace: [Fri Apr 12 11:13:50 2019][2937752.255774] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:50 2019][2937752.261180] [] schedule+0x29/0x70 [Fri Apr 12 11:13:50 2019][2937752.266335] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:50 2019][2937752.273297] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:50 2019][2937752.279075] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:50 2019][2937752.285779] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:50 2019][2937752.290837] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.297109] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:50 2019][2937752.303725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.309994] osp-syn-29-1 S ffff8b8cc2639040 0 2462 2 0x00000080 [Fri Apr 12 11:13:50 2019][2937752.317270] Call Trace: [Fri Apr 12 11:13:50 2019][2937752.319901] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:50 2019][2937752.326000] [] schedule+0x29/0x70 [Fri Apr 12 11:13:50 2019][2937752.331156] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:50 2019][2937752.338488] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:50 2019][2937752.344263] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.351597] [] ? llog_open+0x140/0x3d0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.358136] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937752.364694] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.371948] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.379191] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.386523] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.394288] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.401535] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.409316] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.416978] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:50 2019][2937752.423423] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:50 2019][2937752.429437] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937752.435987] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.442785] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:50 2019][2937752.449237] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:50 2019][2937752.454908] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:50 2019][2937752.462477] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:50 2019][2937752.467539] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.473828] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:50 2019][2937752.480445] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.486718] osp-pre-29-1 S ffff8b8cc263a080 0 2463 2 0x00000080 [Fri Apr 12 11:13:50 2019][2937752.493995] Call Trace: [Fri Apr 12 11:13:50 2019][2937752.496627] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:50 2019][2937752.502034] [] schedule+0x29/0x70 [Fri Apr 12 11:13:50 2019][2937752.507188] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:50 2019][2937752.514150] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:50 2019][2937752.519909] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:50 2019][2937752.526611] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:50 2019][2937752.531673] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.535159] LustreError: dumping log to /tmp/lustre-log.1555092830.118430 [Fri Apr 12 11:13:50 2019][2937752.544907] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:50 2019][2937752.551529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.557802] osp-syn-1-1 S ffff8b8cc263b0c0 0 2464 2 0x00000080 [Fri Apr 12 11:13:50 2019][2937752.565071] Call Trace: [Fri Apr 12 11:13:50 2019][2937752.567702] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:50 2019][2937752.573804] [] schedule+0x29/0x70 [Fri Apr 12 11:13:50 2019][2937752.578957] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:50 2019][2937752.586268] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:50 2019][2937752.592058] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.599374] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:50 2019][2937752.605044] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937752.611594] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.618838] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.626086] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.633417] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.641183] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.648430] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.656193] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.663853] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:50 2019][2937752.670300] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:50 2019][2937752.676319] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937752.682872] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.689666] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:50 2019][2937752.696116] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:50 2019][2937752.701784] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:50 2019][2937752.709353] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:50 2019][2937752.714413] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.720682] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:50 2019][2937752.727317] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.733584] osp-pre-1-1 S ffff8b8cc263c100 0 2465 2 0x00000080 [Fri Apr 12 11:13:50 2019][2937752.740851] Call Trace: [Fri Apr 12 11:13:50 2019][2937752.743484] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:50 2019][2937752.748893] [] schedule+0x29/0x70 [Fri Apr 12 11:13:50 2019][2937752.754045] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:50 2019][2937752.761008] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:50 2019][2937752.766767] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:50 2019][2937752.773471] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:50 2019][2937752.778532] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.784807] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:50 2019][2937752.791427] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:50 2019][2937752.797702] osp-syn-22-1 S ffff8b8cc263d140 0 2466 2 0x00000080 [Fri Apr 12 11:13:50 2019][2937752.804980] Call Trace: [Fri Apr 12 11:13:50 2019][2937752.807608] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:50 2019][2937752.813702] [] schedule+0x29/0x70 [Fri Apr 12 11:13:50 2019][2937752.818857] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:50 2019][2937752.826170] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:50 2019][2937752.831945] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.839291] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:50 2019][2937752.844982] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:50 2019][2937752.851541] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.858792] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:50 2019][2937752.866039] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937752.873383] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937752.881152] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937752.888404] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937752.896171] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937752.903832] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:51 2019][2937752.910279] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:51 2019][2937752.916303] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:51 2019][2937752.922889] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:51 2019][2937752.929704] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:51 2019][2937752.936163] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:51 2019][2937752.941846] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:51 2019][2937752.949464] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:51 2019][2937752.954535] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937752.960811] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:51 2019][2937752.967437] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937752.973720] osp-pre-22-1 S ffff8b8cc263e180 0 2467 2 0x00000080 [Fri Apr 12 11:13:51 2019][2937752.981030] Call Trace: [Fri Apr 12 11:13:51 2019][2937752.983663] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:51 2019][2937752.989069] [] schedule+0x29/0x70 [Fri Apr 12 11:13:51 2019][2937752.994224] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:51 2019][2937753.001197] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:51 2019][2937753.006951] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:51 2019][2937753.013655] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:51 2019][2937753.018716] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.024992] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:51 2019][2937753.031612] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.037878] osp-syn-3-1 S ffff8b8be6b20000 0 2468 2 0x00000080 [Fri Apr 12 11:13:51 2019][2937753.045154] Call Trace: [Fri Apr 12 11:13:51 2019][2937753.047788] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:51 2019][2937753.053887] [] schedule+0x29/0x70 [Fri Apr 12 11:13:51 2019][2937753.059040] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:51 2019][2937753.066351] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:51 2019][2937753.072130] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.079449] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:51 2019][2937753.086005] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.093260] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.100542] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.107875] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:51 2019][2937753.114865] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.122636] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.129884] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.137656] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.145319] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:51 2019][2937753.151767] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:51 2019][2937753.157810] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:51 2019][2937753.164369] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.171176] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:51 2019][2937753.177627] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:51 2019][2937753.183295] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:51 2019][2937753.190868] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:51 2019][2937753.195927] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.202204] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:51 2019][2937753.208827] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.215099] osp-pre-3-1 S ffff8b8be6b21040 0 2469 2 0x00000080 [Fri Apr 12 11:13:51 2019][2937753.222377] Call Trace: [Fri Apr 12 11:13:51 2019][2937753.225008] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:51 2019][2937753.230415] [] schedule+0x29/0x70 [Fri Apr 12 11:13:51 2019][2937753.235567] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:51 2019][2937753.242531] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:51 2019][2937753.248290] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:51 2019][2937753.254991] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:51 2019][2937753.260053] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.266321] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:51 2019][2937753.272940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.279207] osp-syn-27-1 S ffff8b8be6b22080 0 2470 2 0x00000080 [Fri Apr 12 11:13:51 2019][2937753.286475] Call Trace: [Fri Apr 12 11:13:51 2019][2937753.289105] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:51 2019][2937753.295207] [] schedule+0x29/0x70 [Fri Apr 12 11:13:51 2019][2937753.300362] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:51 2019][2937753.307670] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:51 2019][2937753.313446] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.320762] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:51 2019][2937753.327309] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.334557] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.341803] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.349139] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.356928] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.364174] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.371937] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.379599] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:51 2019][2937753.386055] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:51 2019][2937753.392074] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:51 2019][2937753.398623] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.405421] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:51 2019][2937753.411865] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:51 2019][2937753.417535] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:51 2019][2937753.425106] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:51 2019][2937753.430167] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.436443] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:51 2019][2937753.443062] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.449328] osp-pre-27-1 S ffff8b8be6b230c0 0 2471 2 0x00000080 [Fri Apr 12 11:13:51 2019][2937753.456598] Call Trace: [Fri Apr 12 11:13:51 2019][2937753.459228] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:51 2019][2937753.464635] [] schedule+0x29/0x70 [Fri Apr 12 11:13:51 2019][2937753.469788] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:51 2019][2937753.476754] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:51 2019][2937753.482512] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:51 2019][2937753.489215] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:51 2019][2937753.494275] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.500551] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:51 2019][2937753.507173] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.513445] osp-syn-37-1 S ffff8b8be6b24100 0 2472 2 0x00000080 [Fri Apr 12 11:13:51 2019][2937753.520715] Call Trace: [Fri Apr 12 11:13:51 2019][2937753.523344] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:51 2019][2937753.529446] [] schedule+0x29/0x70 [Fri Apr 12 11:13:51 2019][2937753.534600] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:51 2019][2937753.541909] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:51 2019][2937753.547687] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.554998] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:51 2019][2937753.560669] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:51 2019][2937753.567218] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.574463] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.581708] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.589041] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.596810] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.604053] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.611821] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.619480] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:51 2019][2937753.625927] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:51 2019][2937753.631946] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:51 2019][2937753.638536] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.645339] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:51 2019][2937753.651795] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:51 2019][2937753.657470] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:51 2019][2937753.665043] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:51 2019][2937753.670104] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.676409] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:51 2019][2937753.683023] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.689298] osp-pre-37-1 S ffff8b8be6b25140 0 2473 2 0x00000080 [Fri Apr 12 11:13:51 2019][2937753.696574] Call Trace: [Fri Apr 12 11:13:51 2019][2937753.699208] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:51 2019][2937753.704613] [] schedule+0x29/0x70 [Fri Apr 12 11:13:51 2019][2937753.709767] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:51 2019][2937753.716730] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:51 2019][2937753.722490] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:51 2019][2937753.729194] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:51 2019][2937753.734274] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.740548] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:51 2019][2937753.747171] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:51 2019][2937753.753440] osp-syn-36-1 S ffff8b8be6b26180 0 2474 2 0x00000080 [Fri Apr 12 11:13:51 2019][2937753.760717] Call Trace: [Fri Apr 12 11:13:51 2019][2937753.763350] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:51 2019][2937753.769449] [] schedule+0x29/0x70 [Fri Apr 12 11:13:51 2019][2937753.774606] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:51 2019][2937753.781928] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:51 2019][2937753.787824] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.795152] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:51 2019][2937753.800823] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:51 2019][2937753.807383] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.814645] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.821898] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.829217] [] ? osd_key_init+0x25/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:51 2019][2937753.836221] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.844005] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.851260] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.859025] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:51 2019][2937753.866708] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:51 2019][2937753.873160] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:52 2019][2937753.879204] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937753.885776] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:52 2019][2937753.892605] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:52 2019][2937753.899068] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:52 2019][2937753.904754] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:52 2019][2937753.912328] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:52 2019][2937753.917387] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937753.923662] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:52 2019][2937753.930294] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937753.936574] osp-pre-36-1 S ffff8b8a27e70000 0 2475 2 0x00000080 [Fri Apr 12 11:13:52 2019][2937753.943860] Call Trace: [Fri Apr 12 11:13:52 2019][2937753.946503] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:52 2019][2937753.951919] [] schedule+0x29/0x70 [Fri Apr 12 11:13:52 2019][2937753.957071] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:52 2019][2937753.964053] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:52 2019][2937753.969826] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:52 2019][2937753.976548] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:52 2019][2937753.981613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937753.987914] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:52 2019][2937753.994540] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.000811] osp-syn-8-1 S ffff8b8a27e71040 0 2476 2 0x00000080 [Fri Apr 12 11:13:52 2019][2937754.008090] Call Trace: [Fri Apr 12 11:13:52 2019][2937754.010731] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:52 2019][2937754.016850] [] schedule+0x29/0x70 [Fri Apr 12 11:13:52 2019][2937754.022000] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:52 2019][2937754.029320] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:52 2019][2937754.035095] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.042434] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:52 2019][2937754.048106] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937754.054665] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.061926] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.069183] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.076510] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:52 2019][2937754.083500] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.091289] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.098552] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.106314] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.113976] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:52 2019][2937754.120452] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:52 2019][2937754.126480] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937754.133038] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.139838] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:52 2019][2937754.146290] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:52 2019][2937754.151974] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:52 2019][2937754.159547] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:52 2019][2937754.164605] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.170881] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:52 2019][2937754.177501] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.183788] osp-pre-8-1 S ffff8b8a27e72080 0 2477 2 0x00000080 [Fri Apr 12 11:13:52 2019][2937754.191063] Call Trace: [Fri Apr 12 11:13:52 2019][2937754.193695] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:52 2019][2937754.199100] [] schedule+0x29/0x70 [Fri Apr 12 11:13:52 2019][2937754.204255] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:52 2019][2937754.211218] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:52 2019][2937754.216974] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:52 2019][2937754.223680] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:52 2019][2937754.228740] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.235016] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:52 2019][2937754.241637] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.247911] osp-syn-35-1 S ffff8b8a27e730c0 0 2478 2 0x00000080 [Fri Apr 12 11:13:52 2019][2937754.255188] Call Trace: [Fri Apr 12 11:13:52 2019][2937754.257817] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:52 2019][2937754.263910] [] schedule+0x29/0x70 [Fri Apr 12 11:13:52 2019][2937754.269057] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:52 2019][2937754.276365] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:52 2019][2937754.282145] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.289466] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937754.296015] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.303262] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.310507] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.317839] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.325605] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.332852] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.340614] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.348275] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:52 2019][2937754.354723] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:52 2019][2937754.360743] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937754.367294] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.374091] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:52 2019][2937754.380533] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:52 2019][2937754.386205] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:52 2019][2937754.393777] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:52 2019][2937754.398836] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.405110] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:52 2019][2937754.411725] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.417998] osp-pre-35-1 S ffff8b8a27e74100 0 2479 2 0x00000080 [Fri Apr 12 11:13:52 2019][2937754.425268] Call Trace: [Fri Apr 12 11:13:52 2019][2937754.427898] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:52 2019][2937754.433305] [] schedule+0x29/0x70 [Fri Apr 12 11:13:52 2019][2937754.438459] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:52 2019][2937754.445430] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:52 2019][2937754.451190] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:52 2019][2937754.457891] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:52 2019][2937754.462947] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.469222] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:52 2019][2937754.475843] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.482115] osp-syn-28-1 S ffff8b8a27e75140 0 2480 2 0x00000080 [Fri Apr 12 11:13:52 2019][2937754.489384] Call Trace: [Fri Apr 12 11:13:52 2019][2937754.492014] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:52 2019][2937754.498115] [] schedule+0x29/0x70 [Fri Apr 12 11:13:52 2019][2937754.503269] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:52 2019][2937754.510579] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:52 2019][2937754.516353] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.523670] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937754.530221] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.537470] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.544722] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.552046] [] ? osd_key_init+0x75/0x500 [osd_ldiskfs] [Fri Apr 12 11:13:52 2019][2937754.559033] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.566815] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.574089] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.581876] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.589563] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:52 2019][2937754.596014] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:52 2019][2937754.602166] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937754.608772] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.615569] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:52 2019][2937754.622019] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:52 2019][2937754.627712] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:52 2019][2937754.635289] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:52 2019][2937754.640346] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.646613] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:52 2019][2937754.653239] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.659512] osp-pre-28-1 S ffff8b8a27e76180 0 2481 2 0x00000080 [Fri Apr 12 11:13:52 2019][2937754.666787] Call Trace: [Fri Apr 12 11:13:52 2019][2937754.669420] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:52 2019][2937754.674828] [] schedule+0x29/0x70 [Fri Apr 12 11:13:52 2019][2937754.679983] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:52 2019][2937754.686973] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:52 2019][2937754.692731] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:52 2019][2937754.699450] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:52 2019][2937754.704550] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.710830] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:52 2019][2937754.717449] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:52 2019][2937754.723725] osp-syn-30-1 S ffff8b8a7dbe8000 0 2482 2 0x00000080 [Fri Apr 12 11:13:52 2019][2937754.731001] Call Trace: [Fri Apr 12 11:13:52 2019][2937754.733646] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:52 2019][2937754.739755] [] schedule+0x29/0x70 [Fri Apr 12 11:13:52 2019][2937754.744933] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:52 2019][2937754.752246] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:52 2019][2937754.758022] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.765335] [] ? __slab_free+0x81/0x2f0 [Fri Apr 12 11:13:52 2019][2937754.771013] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937754.777576] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.784846] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.792100] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.799452] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.807232] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.814485] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.822253] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.829917] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:52 2019][2937754.836369] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:52 2019][2937754.842389] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:52 2019][2937754.848944] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:52 2019][2937754.855747] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:52 2019][2937754.862199] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:52 2019][2937754.867870] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:52 2019][2937754.875452] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937754.880523] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937754.886802] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937754.893433] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937754.899703] osp-pre-30-1 S ffff8b8a7dbe9040 0 2483 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937754.906980] Call Trace: [Fri Apr 12 11:13:53 2019][2937754.909611] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:53 2019][2937754.915020] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937754.920175] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:53 2019][2937754.927144] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937754.932903] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:53 2019][2937754.939625] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937754.944685] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937754.950962] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937754.957581] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937754.963854] osp-syn-34-1 S ffff8b8a7dbea080 0 2484 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937754.971125] Call Trace: [Fri Apr 12 11:13:53 2019][2937754.973754] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:53 2019][2937754.979859] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937754.985019] [] osp_sync_process_queues+0x163f/0x2020 [osp] [Fri Apr 12 11:13:53 2019][2937754.992348] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937754.998130] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.005449] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:53 2019][2937755.012002] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.019257] [] llog_cat_process_cb+0x239/0x250 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.026501] [] llog_process_thread+0x892/0x15a0 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.033842] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.041607] [] llog_process_or_fork+0xbc/0x450 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.048856] [] ? llog_cat_cancel_records+0x3c0/0x3c0 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.056620] [] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.064285] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:13:53 2019][2937755.070726] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:13:53 2019][2937755.076751] [] ? osp_sync_force+0x300/0x300 [osp] [Fri Apr 12 11:13:53 2019][2937755.083307] [] llog_cat_process+0x2e/0x30 [obdclass] [Fri Apr 12 11:13:53 2019][2937755.090105] [] osp_sync_thread+0x21f/0xa70 [osp] [Fri Apr 12 11:13:53 2019][2937755.096559] [] ? __schedule+0x3ff/0x890 [Fri Apr 12 11:13:53 2019][2937755.102232] [] ? osp_sync_process_committed+0x6c0/0x6c0 [osp] [Fri Apr 12 11:13:53 2019][2937755.109808] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.114868] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.121147] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.127776] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.134092] osp-pre-34-1 S ffff8b8a7dbeb0c0 0 2485 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.141369] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.144021] [] ? ktime_get+0x52/0xe0 [Fri Apr 12 11:13:53 2019][2937755.149435] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.154597] [] osp_precreate_thread+0xb8b/0x1380 [osp] [Fri Apr 12 11:13:53 2019][2937755.161571] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.167334] [] ? osp_init_pre_fid+0x600/0x600 [osp] [Fri Apr 12 11:13:53 2019][2937755.174037] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.179101] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.185380] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.192007] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.198305] lquota_wb_fir-M S ffff8b8a2b711040 0 2503 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.205590] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.208223] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.213369] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:13:53 2019][2937755.219401] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:13:53 2019][2937755.225956] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Fri Apr 12 11:13:53 2019][2937755.232754] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Fri Apr 12 11:13:53 2019][2937755.239381] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.245136] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Fri Apr 12 11:13:53 2019][2937755.251512] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.256583] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.262865] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.269497] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.275776] lquota_wb_fir-M S ffff8b75d3f130c0 0 2506 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.283052] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.285686] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.290834] [] schedule_timeout+0x168/0x2d0 [Fri Apr 12 11:13:53 2019][2937755.296846] [] ? __internal_add_timer+0x130/0x130 [Fri Apr 12 11:13:53 2019][2937755.303391] [] ? qsd_job_pending+0x7e/0x1b0 [lquota] [Fri Apr 12 11:13:53 2019][2937755.310209] [] qsd_upd_thread+0xae6/0xe30 [lquota] [Fri Apr 12 11:13:53 2019][2937755.316835] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.322604] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [Fri Apr 12 11:13:53 2019][2937755.329001] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.334080] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.340366] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.346998] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.353287] dist_txn-1 S ffff8b75d3f12080 0 2507 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.360570] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.363201] [] ? recalc_sigpending+0x1b/0x70 [Fri Apr 12 11:13:53 2019][2937755.369302] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.374518] [] distribute_txn_commit_thread+0x9ac/0xca0 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.382374] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.388187] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.395241] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.400304] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.406578] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.413200] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.419474] mdt_out02_003 S ffff8b86e9342080 0 2516 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.426748] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.429381] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.434576] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.441455] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.447267] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.453767] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.461345] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.466406] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.472692] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.479312] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.485586] mdt_io03_086 S ffff8b86e9345140 0 3056 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.492871] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.495520] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.500724] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.507611] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.513411] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.519897] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.527495] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.532579] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.538868] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.545491] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.551783] mdt_io03_087 S ffff8b8a2b7130c0 0 3057 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.559058] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.561701] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.566916] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.573812] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.579637] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.586122] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.593701] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.598755] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.605036] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.611675] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.617947] mdt_io03_088 S ffff8b8a2b712080 0 3058 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.625249] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.627899] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.633097] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.639981] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.645790] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.652286] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.659869] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.664940] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.671243] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.677870] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.684151] mdt_io03_089 S ffff8b8a2b714100 0 3059 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.691429] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.694084] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.699310] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.706205] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.712006] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.718494] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.726070] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.731136] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.737413] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.744041] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.750322] mdt_io03_090 S ffff8b8a2b715140 0 3060 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.757601] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.760241] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.765431] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.772315] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.778107] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.784599] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.792187] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:53 2019][2937755.797247] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.803522] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:53 2019][2937755.810142] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:53 2019][2937755.816413] mdt_io03_091 S ffff8b8a2b716180 0 3061 2 0x00000080 [Fri Apr 12 11:13:53 2019][2937755.823685] Call Trace: [Fri Apr 12 11:13:53 2019][2937755.826317] [] schedule+0x29/0x70 [Fri Apr 12 11:13:53 2019][2937755.831514] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.838422] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:53 2019][2937755.844223] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:53 2019][2937755.850724] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937755.858297] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937755.863355] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937755.869639] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937755.876273] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937755.882832] mdt_io03_092 S ffff8b73f7e42080 0 3062 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937755.890145] Call Trace: [Fri Apr 12 11:13:54 2019][2937755.892784] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937755.897977] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937755.904875] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937755.910667] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937755.917189] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937755.924790] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937755.929852] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937755.936135] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937755.942756] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937755.949038] mdt_io03_093 S ffff8b73f7e40000 0 3064 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937755.956321] Call Trace: [Fri Apr 12 11:13:54 2019][2937755.958957] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937755.964159] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937755.971084] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937755.976883] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937755.983370] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937755.990965] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937755.996040] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.002315] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.008952] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.015224] mdt_io03_094 S ffff8b73f7e430c0 0 3065 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.022503] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.025131] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.030340] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.037224] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.043016] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.049496] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.057068] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.062131] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.068407] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.075026] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.081300] mdt_io03_095 S ffff8b73f7e45140 0 3066 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.088577] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.091209] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.096397] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.103276] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.109083] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.115582] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.123159] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.128219] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.134510] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.141129] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.147404] mdt_io03_096 S ffff8b84b8f70000 0 3067 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.154696] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.157350] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.162539] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.169418] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.175215] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.181699] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.189274] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.194336] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.200613] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.207252] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.213545] mdt_io03_097 S ffff8b84b8f71040 0 3068 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.220816] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.223452] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.229149] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.236037] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.241837] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.248323] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.255897] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.260955] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.267232] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.273873] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.280145] mdt_io03_098 S ffff8b6315fb1040 0 3070 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.287447] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.290084] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.295321] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.302207] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.308000] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.314487] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.322059] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.327120] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.333415] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.340202] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.346509] mdt_io03_099 S ffff8b6315fb5140 0 3072 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.353793] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.356425] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.361618] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.368499] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.374287] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.380769] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.388347] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.393436] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.399725] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.406345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.412619] kworker/28:2 S ffff8b58d53d8000 0 5813 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.419902] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.422533] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:54 2019][2937756.428726] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.433881] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:54 2019][2937756.439649] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:54 2019][2937756.446361] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.451435] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.457729] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.464354] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.470661] ldlm_cn02_022 S ffff8b7cf3251040 0 6477 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.477945] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.480578] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.485773] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.492670] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.498465] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.504963] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.512552] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.517612] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.523884] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.530511] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.536781] ldlm_cn02_023 S ffff8b8cf6ddb0c0 0 6478 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.544057] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.546689] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.551882] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.558760] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.564550] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.571040] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.578621] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.583679] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.590015] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.596635] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.602908] ldlm_cn02_024 S ffff8b8cf6dde180 0 6479 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.610184] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.612822] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.618015] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.624896] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.630691] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.637173] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.644757] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.649824] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.656109] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.662733] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.669135] ldlm_cn02_025 S ffff8b8cf6ddd140 0 6480 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.676419] Call Trace: [Fri Apr 12 11:13:54 2019][2937756.679060] [] schedule+0x29/0x70 [Fri Apr 12 11:13:54 2019][2937756.684289] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.691186] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:54 2019][2937756.697011] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.703500] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:54 2019][2937756.711081] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:54 2019][2937756.716140] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.722426] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:54 2019][2937756.729050] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:54 2019][2937756.735330] ldlm_cn02_026 S ffff8b8cf6ddc100 0 6481 2 0x00000080 [Fri Apr 12 11:13:54 2019][2937756.742736] Call Trace: [Fri Apr 12 11:13:55 2019][2937756.745387] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937756.750622] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.757536] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937756.763346] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.769832] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.777418] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937756.782479] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937756.788773] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937756.795392] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937756.801663] ldlm_cn00_025 S ffff8b88d036d140 0 6482 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937756.808932] Call Trace: [Fri Apr 12 11:13:55 2019][2937756.811563] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937756.816753] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.823639] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937756.829449] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.835957] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.843547] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937756.848608] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937756.854893] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937756.861510] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937756.867784] ldlm_cn00_026 S ffff8b8cf1ffa080 0 6483 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937756.875097] Call Trace: [Fri Apr 12 11:13:55 2019][2937756.877740] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937756.882958] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.889847] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937756.895655] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.902152] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.909768] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937756.914853] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937756.921171] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937756.927834] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937756.934131] ldlm_cn00_027 S ffff8b8cf1ffd140 0 6484 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937756.941432] Call Trace: [Fri Apr 12 11:13:55 2019][2937756.944064] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937756.949264] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.956158] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937756.961949] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.968439] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937756.976020] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937756.981078] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937756.987388] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937756.994032] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.000301] ldlm_cn00_028 S ffff8b8cf1ffc100 0 6485 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.007585] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.010218] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.015413] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.022291] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.028084] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.034563] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.042158] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.047217] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.053501] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.060131] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.066439] ldlm_cn00_029 S ffff8b8cf1ff9040 0 6486 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.074013] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.076808] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.082010] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.088897] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.094693] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.101179] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.108757] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.113821] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.120098] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.126728] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.133000] ldlm_cn00_030 S ffff8b8cf1ffb0c0 0 6487 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.140296] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.142938] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.148138] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.155032] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.160828] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.167326] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.174911] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.179978] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.186260] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.192879] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.199165] ldlm_cn00_031 S ffff8b8cf1ffe180 0 6488 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.206475] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.209118] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.214319] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.221224] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.227029] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.233515] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.241097] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.246159] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.252432] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.259054] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.265325] ldlm_cn02_027 S ffff8b8b083b8000 0 6493 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.272603] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.275252] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.280443] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.287323] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.293119] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.299616] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.307189] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.312259] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.318544] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.325160] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.331434] ldlm_cn02_028 R running task 0 6494 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.338711] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.341343] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.346537] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.353427] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.359233] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.365727] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.373351] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.378412] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.384689] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.391309] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.397584] ldlm_cn02_029 S ffff8b8cf4734100 0 6495 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.404893] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.407526] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.412715] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.419598] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.425391] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.431882] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.439463] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.444538] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.450820] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.457439] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.463711] ldlm_cn02_030 S ffff8b8cf47330c0 0 6496 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.470996] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.473639] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.478830] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.485713] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.491511] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.498006] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.505582] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.510640] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.516943] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.523564] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.529839] ldlm_cn02_031 S ffff8b895eef9040 0 6497 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.537131] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.539767] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.544958] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.551837] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.557631] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.564121] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.571704] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.576774] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.583058] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.589685] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.595967] kworker/14:0 S ffff8b6ef2748000 0 6950 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.603249] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.605887] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.611036] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:55 2019][2937757.616804] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:55 2019][2937757.623520] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.628581] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.634868] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.641486] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.647796] mdt_out00_004 S ffff8b8b0a71d140 0 8935 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.655098] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.657732] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.662941] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.669818] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.675616] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.682100] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.689704] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.694762] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.701037] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.707656] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.713929] mdt_out00_005 S ffff8b6a043ce180 0 9264 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.721200] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.723834] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.729034] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.735921] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:55 2019][2937757.741714] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.748193] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:55 2019][2937757.755767] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.760826] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.767104] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.773726] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.780004] kworker/8:1 S ffff8b5c7afcd140 0 15011 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.787283] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.789924] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.795074] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:55 2019][2937757.800828] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:55 2019][2937757.807539] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.812597] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.818883] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:55 2019][2937757.825503] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:55 2019][2937757.831775] kworker/42:0 S ffff8b782d21e180 0 16497 2 0x00000080 [Fri Apr 12 11:13:55 2019][2937757.839066] Call Trace: [Fri Apr 12 11:13:55 2019][2937757.841710] [] schedule+0x29/0x70 [Fri Apr 12 11:13:55 2019][2937757.846879] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:55 2019][2937757.852639] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:55 2019][2937757.859344] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:55 2019][2937757.864404] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937757.870679] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937757.877304] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937757.883574] ldlm_cn03_017 S ffff8b681ffeb0c0 0 18903 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937757.890853] Call Trace: [Fri Apr 12 11:13:56 2019][2937757.893504] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937757.898694] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937757.905587] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:56 2019][2937757.911406] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937757.917909] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937757.925483] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937757.930566] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937757.936848] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937757.943476] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937757.949768] ldlm_cn03_018 S ffff8b5c7afc9040 0 18904 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937757.957060] Call Trace: [Fri Apr 12 11:13:56 2019][2937757.959701] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937757.964890] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937757.971776] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:56 2019][2937757.977573] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937757.984056] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937757.991646] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937757.996704] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.002981] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.009626] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.015904] ldlm_cn03_019 S ffff8b5c7afc8000 0 18905 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.023205] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.025848] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.031040] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937758.037925] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:56 2019][2937758.043720] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937758.050245] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:56 2019][2937758.057833] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.062905] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.069178] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.075798] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.082071] kworker/20:0 S ffff8b531563d140 0 20142 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.089366] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.092000] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:56 2019][2937758.098188] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.103335] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.109100] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.115895] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.120954] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.127249] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.133866] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.140139] kworker/17:0 S ffff8b5f8eef9040 0 24866 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.147414] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.150048] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:56 2019][2937758.156238] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.161386] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.167158] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.173866] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.178928] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.185213] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.191931] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.198211] kworker/46:1 S ffff8b7cfebd30c0 0 24867 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.205490] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.208127] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.213277] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.219030] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.225737] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.230803] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.237075] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.243694] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.249997] kworker/0:0 S ffff8b5cf660d140 0 24961 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.257298] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.259948] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.265094] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.270857] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.277565] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.282623] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.288904] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.295543] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.301821] kworker/1:0 S ffff8b6a82fc30c0 0 25476 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.309131] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.311768] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.316934] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.322711] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.329417] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.334482] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.340757] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.347377] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.353650] kworker/37:0 S ffff8b6cd512d140 0 26717 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.360950] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.363585] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.368734] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.374486] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.381193] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.386253] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.392534] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.399161] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.405443] kworker/40:0 S ffff8b5cfbdb8000 0 26850 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.412734] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.415369] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:56 2019][2937758.421560] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.426704] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.432467] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.439175] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.444235] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.450521] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.457162] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.463452] kworker/39:1 S ffff8b84b8f72080 0 27263 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.470760] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.473397] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:56 2019][2937758.479609] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.484774] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.490538] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.497245] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.502305] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.508590] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.515214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.521493] kworker/11:1 S ffff8b82cbf830c0 0 27914 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.528802] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.531445] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.536596] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.542347] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.549194] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.554263] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.560540] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.567161] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.573432] kworker/3:0 S ffff8b82cbf85140 0 27935 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.580733] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.583378] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.588526] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.594280] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.600994] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.606056] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.612342] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.619088] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.625477] kworker/13:0 S ffff8b6cd512a080 0 29284 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.632871] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.635538] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:56 2019][2937758.641821] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.647087] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.652928] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.659724] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.664894] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.671316] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.677986] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.684300] kworker/34:2 S ffff8b7cf3256180 0 29749 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.691580] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.694214] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.699363] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.705121] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.711850] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.716915] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.723190] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.729820] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.736101] kworker/5:0 S ffff8b67cbe5a080 0 29823 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.743385] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.746021] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.751167] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.756919] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.763629] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.768689] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.774978] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.781616] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.787895] kworker/36:1 S ffff8b5965d88000 0 30971 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.795188] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.797824] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:56 2019][2937758.804019] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.809192] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:56 2019][2937758.814944] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:56 2019][2937758.821671] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:56 2019][2937758.826734] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.833009] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:56 2019][2937758.839626] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:56 2019][2937758.845920] kworker/36:2 S ffff8b5af4e5a080 0 32026 2 0x00000080 [Fri Apr 12 11:13:56 2019][2937758.853202] Call Trace: [Fri Apr 12 11:13:56 2019][2937758.855834] [] schedule+0x29/0x70 [Fri Apr 12 11:13:56 2019][2937758.860983] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937758.866742] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937758.873446] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937758.878529] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937758.884821] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937758.891456] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937758.897730] kworker/9:1 S ffff8b681ffec100 0 32709 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937758.905011] Call Trace: [Fri Apr 12 11:13:57 2019][2937758.907647] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937758.912794] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937758.918546] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937758.925254] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937758.930315] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937758.936590] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937758.943212] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937758.949487] kworker/28:1 S ffff8b5965d8c100 0 33122 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937758.956768] Call Trace: [Fri Apr 12 11:13:57 2019][2937758.959402] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937758.964551] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937758.970306] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937758.977013] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937758.982075] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937758.988362] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937758.995003] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.001277] kworker/32:2 S ffff8b5cfbdbe180 0 33497 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.008554] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.011200] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.016345] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.022101] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.028817] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.033875] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.040144] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.046763] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.053039] kworker/u257:2 S ffff8b5c7afcc100 0 34641 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.060322] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.062954] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:57 2019][2937759.069142] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.074303] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.080057] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.086765] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.091834] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.098120] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.104739] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.111031] kworker/8:0 S ffff8b5383238000 0 34751 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.118319] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.120958] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:57 2019][2937759.127163] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.132331] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.138091] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.144809] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.149871] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.156145] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.162761] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.169130] kworker/15:2 S ffff8b8cf729a080 0 35529 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.176527] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.179164] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.184339] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.190091] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.196799] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.201861] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.208154] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.214781] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.221057] kworker/44:2 S ffff8b5315639040 0 36816 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.228347] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.230982] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.236150] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.241900] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.248609] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.253667] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.259942] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.266567] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.272843] kworker/12:1 S ffff8b51eb171040 0 37779 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.280156] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.282827] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:57 2019][2937759.289035] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.294198] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.299954] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.306660] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.311730] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.318008] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.324625] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.330901] kworker/35:0 S ffff8b8c8dec2080 0 38160 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.338225] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.340862] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.346027] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.351790] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.358496] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.363557] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.369831] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.376452] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.382727] kworker/27:0 S ffff8b84b8f76180 0 38262 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.390008] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.392641] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.397794] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.403563] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.410271] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.415332] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.421605] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.428226] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.434492] kworker/43:0 S ffff8b8c19b86180 0 39820 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.441767] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.444400] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.449551] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.455317] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.462019] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.467080] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.473356] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.479978] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.486263] kworker/u259:1 S ffff8b7cf32530c0 0 40303 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.493549] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.496185] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:57 2019][2937759.502372] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.507525] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.513307] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.520013] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.525072] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.531348] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.537971] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.544246] kworker/u257:1 S ffff8b5778b80000 0 40885 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.551551] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.554188] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:57 2019][2937759.560376] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.565520] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.571295] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.578011] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.583071] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.589345] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.595965] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.602239] kworker/16:1 S ffff8b51eb1730c0 0 40977 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.609516] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.612147] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.617297] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.623049] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.629768] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.634830] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.641105] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.647727] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.654000] kworker/u260:2 S ffff8b7da95230c0 0 41531 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.661292] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.663928] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:57 2019][2937759.670123] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.675279] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.681056] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.687796] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.692854] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.699142] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.705772] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.712053] kworker/21:1 S ffff8b5f8eefc100 0 42367 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.719333] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.721968] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.727116] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.732867] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.739576] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.744642] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.750934] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.757562] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.763835] kworker/20:2 S ffff8b4f5b691040 0 42444 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.771144] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.773780] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.778927] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.784679] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.791392] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.796483] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.802760] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.809399] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.815671] kworker/24:0 S ffff8b51eb172080 0 43310 2 0x00000080 [Fri Apr 12 11:13:57 2019][2937759.822952] Call Trace: [Fri Apr 12 11:13:57 2019][2937759.825587] [] schedule+0x29/0x70 [Fri Apr 12 11:13:57 2019][2937759.830738] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:57 2019][2937759.836493] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:57 2019][2937759.843209] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:57 2019][2937759.848268] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:57 2019][2937759.854543] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:57 2019][2937759.861194] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937759.867483] kworker/17:2 S ffff8b67cbe5e180 0 43376 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937759.874783] Call Trace: [Fri Apr 12 11:13:58 2019][2937759.877416] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937759.882563] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:58 2019][2937759.888318] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:58 2019][2937759.895025] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937759.900089] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937759.906363] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937759.912983] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937759.919283] kworker/22:1 S ffff8b75e970c100 0 43552 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937759.926565] Call Trace: [Fri Apr 12 11:13:58 2019][2937759.929200] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:58 2019][2937759.935387] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937759.940533] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:58 2019][2937759.946288] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:58 2019][2937759.952997] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937759.958057] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937759.964324] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937759.970947] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937759.977235] kworker/19:2 S ffff8b8c19b85140 0 43802 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937759.984510] Call Trace: [Fri Apr 12 11:13:58 2019][2937759.987147] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:58 2019][2937759.993333] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937759.998479] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:58 2019][2937760.004227] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:58 2019][2937760.010936] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.015998] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.022272] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.028894] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.035181] kworker/44:1 S ffff8b54d397e180 0 43922 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.042456] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.045092] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:58 2019][2937760.051281] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.056427] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:58 2019][2937760.062181] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:58 2019][2937760.068895] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.073953] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.080231] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.086924] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.093373] ldlm_cn03_020 S ffff8b7450658000 0 44377 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.100642] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.103276] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.108473] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.115376] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.121170] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.127669] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.135247] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.140307] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.146594] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.153214] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.159487] ldlm_cn03_021 S ffff8b745065d140 0 44378 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.166771] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.169405] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.174597] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.181481] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.187294] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.193787] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.201360] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.206428] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.212722] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.219345] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.225623] ldlm_cn03_022 S ffff8b745065b0c0 0 44379 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.232916] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.235549] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.240743] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.247644] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.253460] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.259969] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.267549] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.272606] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.278893] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.285515] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.291785] ldlm_cn03_023 S ffff8b745065c100 0 44380 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.299074] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.301724] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.306925] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.313814] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.319624] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.326112] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.333728] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.338792] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.345063] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.351694] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.358002] ldlm_cn03_024 S ffff8b7450659040 0 44381 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.365277] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.367912] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.373102] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.379980] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.385775] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.392253] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.399839] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.404898] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.411175] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.417794] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.424066] ldlm_cn03_025 S ffff8b8c5fd5d140 0 44382 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.431352] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.433984] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.439190] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.446078] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.451871] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.458358] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.465932] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.471018] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.477296] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.483916] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.490190] ldlm_cn03_026 S ffff8b8c8dec0000 0 44384 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.497472] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.500111] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.505352] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.512266] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.518065] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.524568] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.532149] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.537347] [] ? finish_task_switch+0x54/0x1c0 [Fri Apr 12 11:13:58 2019][2937760.543694] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.549986] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.556613] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.562960] kworker/47:0 S ffff8b8b0a71b0c0 0 44759 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.570276] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.572916] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.578067] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:58 2019][2937760.583820] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:58 2019][2937760.590525] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.595597] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.601872] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.608503] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.614774] kworker/39:2 S ffff8b8b0a71c100 0 44858 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.622056] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.624689] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.629867] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:58 2019][2937760.635621] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:58 2019][2937760.642337] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.647396] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.653676] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.660313] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.666616] ldlm_cn03_027 S ffff8b51eb174100 0 45019 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.673932] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.676582] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.681793] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.688681] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.694476] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.700991] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.708578] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.713638] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.719915] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.726535] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.732806] ldlm_cn03_028 S ffff8b5f8eefe180 0 45020 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.740078] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.742719] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.747911] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.754792] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.760581] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.767075] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.774697] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.779757] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.786050] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.792749] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.799093] ldlm_cn03_029 S ffff8b7a6d3cc100 0 45021 2 0x00000080 [Fri Apr 12 11:13:58 2019][2937760.806377] Call Trace: [Fri Apr 12 11:13:58 2019][2937760.809020] [] schedule+0x29/0x70 [Fri Apr 12 11:13:58 2019][2937760.814215] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.821102] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:58 2019][2937760.826905] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.833388] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:58 2019][2937760.840984] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:58 2019][2937760.846043] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.852318] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:58 2019][2937760.858937] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:58 2019][2937760.865212] ldlm_cn03_030 S ffff8b7a6d3ca080 0 45022 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937760.872504] Call Trace: [Fri Apr 12 11:13:59 2019][2937760.875137] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937760.880327] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:59 2019][2937760.887208] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:59 2019][2937760.893001] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:59 2019][2937760.899485] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:59 2019][2937760.907063] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937760.912137] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937760.918411] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937760.925034] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937760.931340] ldlm_cn03_031 S ffff8b54d397d140 0 45023 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937760.938633] Call Trace: [Fri Apr 12 11:13:59 2019][2937760.941268] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937760.946497] [] ptlrpc_wait_event+0x345/0x360 [ptlrpc] [Fri Apr 12 11:13:59 2019][2937760.953408] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:59 2019][2937760.959204] [] ptlrpc_main+0x853/0x1fc0 [ptlrpc] [Fri Apr 12 11:13:59 2019][2937760.965692] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [Fri Apr 12 11:13:59 2019][2937760.973279] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937760.978349] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937760.984627] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937760.991249] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937760.997529] kworker/u259:0 S ffff8b790334b0c0 0 45187 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.004822] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.007453] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:59 2019][2937761.013642] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.018788] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.024542] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.031252] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.036312] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.042591] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.049249] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.055529] kworker/10:1 S ffff8b76fa3e4100 0 45638 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.062819] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.065452] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.070602] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.076354] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.083064] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.088137] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.094418] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.101041] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.107320] kworker/6:0 S ffff8b745f615140 0 45707 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.114612] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.117249] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.122396] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.128150] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.134857] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.139922] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.146195] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.152822] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.159102] kworker/u258:1 S ffff8b681ffee180 0 45718 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.166441] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.169078] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:59 2019][2937761.175284] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.180440] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.186192] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.192927] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.198035] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.204325] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.210945] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.217215] kworker/u256:2 S ffff8b51b2288000 0 45906 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.224532] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.227170] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:59 2019][2937761.233356] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.238513] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.244268] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.250980] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.256038] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.262324] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.268957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.275233] kworker/u258:0 S ffff8b6315fb2080 0 45968 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.282533] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.285168] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:59 2019][2937761.291374] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.296520] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.302290] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.309002] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.314059] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.320337] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.326957] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.333232] kworker/7:2 S ffff8b8ccbe36180 0 46131 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.340540] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.343178] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.348332] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.354086] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.360795] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.365865] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.372139] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.378762] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.385049] pickup S ffff8b67cbe5b0c0 0 46136 19681 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.392327] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.394959] [] ? hrtimer_start_range_ns+0x1ed/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.401666] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.406811] [] schedule_hrtimeout_range_clock+0xb2/0x150 [Fri Apr 12 11:13:59 2019][2937761.413944] [] ? hrtimer_get_res+0x50/0x50 [Fri Apr 12 11:13:59 2019][2937761.419880] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [Fri Apr 12 11:13:59 2019][2937761.427198] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:13:59 2019][2937761.433736] [] ep_poll+0x23e/0x360 [Fri Apr 12 11:13:59 2019][2937761.439003] [] ? wake_up_state+0x20/0x20 [Fri Apr 12 11:13:59 2019][2937761.444757] [] SyS_epoll_wait+0xed/0x120 [Fri Apr 12 11:13:59 2019][2937761.450515] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:13:59 2019][2937761.456709] kworker/u256:0 S ffff8b6315fb6180 0 46263 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.464035] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.466672] [] ? queued_spin_lock_slowpath+0xb/0xf [Fri Apr 12 11:13:59 2019][2937761.473309] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:59 2019][2937761.479496] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.484660] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.490415] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.497136] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.502211] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.508486] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.515109] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.521382] kworker/u260:1 S ffff8b88863230c0 0 46265 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.528664] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.531299] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:59 2019][2937761.537487] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.542634] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.548388] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.555113] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.560172] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.566448] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.573090] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.579361] kworker/23:0 S ffff8b8697f8c100 0 46356 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.586644] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.589297] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.594444] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.600198] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.606904] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.611966] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.618271] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.624969] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.631244] kworker/31:0 S ffff8b888ff530c0 0 46406 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.638559] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.641208] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.646359] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.652121] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.658830] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.663893] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.670177] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.676840] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.683114] kworker/u256:1 S ffff8b89d9ffb0c0 0 46414 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.690407] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.693039] [] ? process_one_work+0x21a/0x440 [Fri Apr 12 11:13:59 2019][2937761.699246] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.704404] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.710164] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.716871] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.721933] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.728208] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.734839] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.741114] kworker/18:2 S ffff8b79ccfab0c0 0 46461 2 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.748438] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.751074] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.756221] [] worker_thread+0x1d9/0x3c0 [Fri Apr 12 11:13:59 2019][2937761.761977] [] ? manage_workers.isra.25+0x2a0/0x2a0 [Fri Apr 12 11:13:59 2019][2937761.768701] [] kthread+0xd1/0xe0 [Fri Apr 12 11:13:59 2019][2937761.773758] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.780040] [] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 12 11:13:59 2019][2937761.786675] [] ? insert_kthread_work+0x40/0x40 [Fri Apr 12 11:13:59 2019][2937761.792949] sshd S ffff8b5af4e5c100 0 46542 19290 0x00000080 [Fri Apr 12 11:13:59 2019][2937761.800226] Call Trace: [Fri Apr 12 11:13:59 2019][2937761.802864] [] ? skb_clone_tx_timestamp+0x25/0xf0 [Fri Apr 12 11:13:59 2019][2937761.809399] [] schedule+0x29/0x70 [Fri Apr 12 11:13:59 2019][2937761.814545] [] schedule_hrtimeout_range_clock+0x12d/0x150 [Fri Apr 12 11:13:59 2019][2937761.821774] [] ? add_wait_queue+0x3c/0x50 [Fri Apr 12 11:13:59 2019][2937761.827616] [] ? tty_write_room+0x20/0x30 [Fri Apr 12 11:13:59 2019][2937761.833987] [] ? n_tty_poll+0x1b7/0x1d0 [Fri Apr 12 11:13:59 2019][2937761.840051] [] schedule_hrtimeout_range+0x13/0x20 [Fri Apr 12 11:13:59 2019][2937761.846598] [] poll_schedule_timeout+0x55/0xb0 [Fri Apr 12 11:13:59 2019][2937761.852882] [] do_select+0x6d1/0x7c0 [Fri Apr 12 11:13:59 2019][2937761.858318] [] ? ip_finish_output+0x284/0x7b0 [Fri Apr 12 11:13:59 2019][2937761.864504] [] ? local_bh_enable+0x17/0x20 [Fri Apr 12 11:13:59 2019][2937761.870431] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:14:00 2019][2937761.875928] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:14:00 2019][2937761.881421] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:14:00 2019][2937761.886918] [] ? __pollwait+0xf0/0xf0 [Fri Apr 12 11:14:00 2019][2937761.892411] [] ? check_preempt_wakeup+0x148/0x250 [Fri Apr 12 11:14:00 2019][2937761.898950] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:14:00 2019][2937761.904710] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:14:00 2019][2937761.911330] [] ? probe_sched_wakeup+0x2b/0xa0 [Fri Apr 12 11:14:00 2019][2937761.917529] [] ? tracing_is_on+0x15/0x30 [Fri Apr 12 11:14:00 2019][2937761.923279] [] ? tracing_record_cmdline+0x1d/0x120 [Fri Apr 12 11:14:00 2019][2937761.929912] [] ? probe_sched_wakeup+0x2b/0xa0 [Fri Apr 12 11:14:00 2019][2937761.936109] [] ? ttwu_do_wakeup+0xb5/0xe0 [Fri Apr 12 11:14:00 2019][2937761.941949] [] ? try_to_wake_up+0x190/0x390 [Fri Apr 12 11:14:00 2019][2937761.947974] [] ? default_wake_function+0x12/0x20 [Fri Apr 12 11:14:00 2019][2937761.954422] [] ? __wake_up_common+0x5b/0x90 [Fri Apr 12 11:14:00 2019][2937761.960449] [] core_sys_select+0x1ff/0x340 [Fri Apr 12 11:14:00 2019][2937761.966375] [] ? __wake_up+0x44/0x50 [Fri Apr 12 11:14:00 2019][2937761.971783] [] ? tty_write_unlock+0x31/0x40 [Fri Apr 12 11:14:00 2019][2937761.977795] [] ? tty_ldisc_deref+0x19/0x20 [Fri Apr 12 11:14:00 2019][2937761.983743] [] ? tty_write+0x1eb/0x2c0 [Fri Apr 12 11:14:00 2019][2937761.989344] [] ? n_tty_ioctl+0xe0/0xe0 [Fri Apr 12 11:14:00 2019][2937761.994933] [] SyS_select+0xca/0x120 [Fri Apr 12 11:14:00 2019][2937762.000340] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:14:00 2019][2937762.006528] bash R running task 0 46544 46542 0x00000088 [Fri Apr 12 11:14:00 2019][2937762.013820] Call Trace: [Fri Apr 12 11:14:00 2019][2937762.016471] [] sched_show_task+0xa8/0x110 [Fri Apr 12 11:14:00 2019][2937762.022310] [] show_state_filter+0x82/0xd0 [Fri Apr 12 11:14:00 2019][2937762.028237] [] sysrq_handle_showstate+0x10/0x20 [Fri Apr 12 11:14:00 2019][2937762.034614] [] __handle_sysrq+0x10d/0x170 [Fri Apr 12 11:14:00 2019][2937762.040457] [] write_sysrq_trigger+0x28/0x40 [Fri Apr 12 11:14:00 2019][2937762.046561] [] proc_reg_write+0x40/0x80 [Fri Apr 12 11:14:00 2019][2937762.052238] [] vfs_write+0xc0/0x1f0 [Fri Apr 12 11:14:00 2019][2937762.057556] [] SyS_write+0x7f/0xf0 [Fri Apr 12 11:14:00 2019][2937762.062793] [] system_call_fastpath+0x22/0x27 [Fri Apr 12 11:14:00 2019][2937762.068982] Sched Debug Version: v0.11, 3.10.0-957.1.3.el7_lustre.x86_64 #1 [Fri Apr 12 11:14:00 2019][2937762.076121] ktime : 2937695516.688118 [Fri Apr 12 11:14:00 2019][2937762.082915] sched_clk : 2937762068.980823 [Fri Apr 12 11:14:00 2019][2937762.089708] cpu_clk : 2937762068.980843 [Fri Apr 12 11:14:00 2019][2937762.096503] jiffies : 7232362841 [Fri Apr 12 11:14:00 2019][2937762.102704] sched_clock_stable() : 1 [Fri Apr 12 11:14:00 2019][2937762.108106] [Fri Apr 12 11:14:00 2019][2937762.109778] sysctl_sched [Fri Apr 12 11:14:00 2019][2937762.112491] .sysctl_sched_latency : 24.000000 [Fri Apr 12 11:14:00 2019][2937762.118857] .sysctl_sched_min_granularity : 10.000000 [Fri Apr 12 11:14:00 2019][2937762.125217] .sysctl_sched_wakeup_granularity : 4.000000 [Fri Apr 12 11:14:00 2019][2937762.131517] .sysctl_sched_child_runs_first : 0 [Fri Apr 12 11:14:00 2019][2937762.137186] .sysctl_sched_features : 56955 [Fri Apr 12 11:14:00 2019][2937762.143199] .sysctl_sched_tunable_scaling : 1 (logaritmic) [Fri Apr 12 11:14:00 2019][2937762.150007] [Fri Apr 12 11:14:00 2019][2937762.151685] cpu#0, 1996.203 MHz [Fri Apr 12 11:14:00 2019][2937762.155034] .nr_running : 0 [Fri Apr 12 11:14:00 2019][2937762.159832] .load : 0 [Fri Apr 12 11:14:00 2019][2937762.164635] .nr_switches : 19427477000 [Fri Apr 12 11:14:00 2019][2937762.170307] .nr_load_updates : 2089349889 [Fri Apr 12 11:14:00 2019][2937762.175884] .nr_uninterruptible : 581971 [Fri Apr 12 11:14:00 2019][2937762.181117] .next_balance : 7232.363032 [Fri Apr 12 11:14:00 2019][2937762.186775] .curr->pid : 118555 [Fri Apr 12 11:14:00 2019][2937762.192002] .clock : 2937762191.337126 [Fri Apr 12 11:14:00 2019][2937762.198184] .cpu_load[0] : 1024 [Fri Apr 12 11:14:00 2019][2937762.203244] .cpu_load[1] : 1798 [Fri Apr 12 11:14:00 2019][2937762.208308] .cpu_load[2] : 2444 [Fri Apr 12 11:14:00 2019][2937762.213369] .cpu_load[3] : 2795 [Fri Apr 12 11:14:00 2019][2937762.218430] .cpu_load[4] : 3255 [Fri Apr 12 11:14:00 2019][2937762.223490] .avg_idle : 123384 [Fri Apr 12 11:14:00 2019][2937762.228726] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:00 2019][2937762.234047] [Fri Apr 12 11:14:00 2019][2937762.235723] cfs_rq[0]:/ [Fri Apr 12 11:14:00 2019][2937762.238348] .exec_clock : 0.000000 [Fri Apr 12 11:14:00 2019][2937762.243759] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:00 2019][2937762.249158] .min_vruntime : 116840947.971981 [Fri Apr 12 11:14:00 2019][2937762.255251] .max_vruntime : 0.000001 [Fri Apr 12 11:14:00 2019][2937762.260650] .spread : 0.000000 [Fri Apr 12 11:14:00 2019][2937762.263501] LustreError: dumping log to /tmp/lustre-log.1555092840.117535 [Fri Apr 12 11:14:00 2019][2937762.273009] .spread0 : 0.000000 [Fri Apr 12 11:14:00 2019][2937762.278409] .nr_spread_over : 0 [Fri Apr 12 11:14:00 2019][2937762.283200] .nr_running : 0 [Fri Apr 12 11:14:00 2019][2937762.287993] .load : 0 [Fri Apr 12 11:14:00 2019][2937762.292803] .runnable_load_avg : 83 [Fri Apr 12 11:14:00 2019][2937762.297682] 35] .tg_load_avg : 0 [Fri Apr 12 11:14:00 2019][2937762.307538] .tg_load_contrib : 0 [Fri Apr 12 11:14:00 2019][2937762.312330] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:00 2019][2937762.317131] .tg->runnable_avg : 0 [Fri Apr 12 11:14:00 2019][2937762.321949] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:00 2019][2937762.326751] .throttled : 0 [Fri Apr 12 11:14:00 2019][2937762.331544] .throttle_count : 0 [Fri Apr 12 11:14:00 2019][2937762.336339] .avg->runnable_avg_sum : 13872 [Fri Apr 12 11:14:00 2019][2937762.341485] .avg->runnable_avg_period : 47029 [Fri Apr 12 11:14:00 2019][2937762.346632] [Fri Apr 12 11:14:00 2019][2937762.348306] rt_rq[0]:/ [Fri Apr 12 11:14:00 2019][2937762.350846] .rt_nr_running : 0 [Fri Apr 12 11:14:00 2019][2937762.355637] .rt_throttled : 0 [Fri Apr 12 11:14:00 2019][2937762.360438] .rt_time : 0.000000 [Fri Apr 12 11:14:00 2019][2937762.365838] .rt_runtime : 837.265973 [Fri Apr 12 11:14:00 2019][2937762.371410] [Fri Apr 12 11:14:00 2019][2937762.373084] runnable tasks: [Fri Apr 12 11:14:00 2019][2937762.376057] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:00 2019][2937762.387019] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:00 2019][2937762.397880] ksoftirqd/0 3 116840939.747581 30726255 120 0.000000 437351.013080 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.409216] kworker/0:0H 5 7736.282938 8 100 0.000000 0.028816 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.420551] migration/0 8 0.000000 27862 0 0.000000 370221.732781 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.431889] rcu_bh 9 513.366593 2 120 0.000000 0.002245 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.443225] lru-add-drain 11 517.368415 2 100 0.000000 0.002755 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.454559] watchdog/0 12 0.000000 734427 0 0.000000 36946.243725 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.465928] irq/26-AMD-Vi 269 0.000000 3 49 0.000000 0.037591 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.477267] kswapd0 277 3055.673452 4 120 0.000000 0.052701 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.488612] kworker/u262:0 7395 6941.455674 4 100 0.000000 0.015670 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.499952] kworker/0:1H 7450 116840956.589383 23665168 100 0.000000 150893.363662 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.511321] dsm_sa_datamgrd 19151 116840567.812142 12196810 120 0.000000 790937.588379 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.522674] kworker/0:2 64678 116840935.982190 581780 120 0.000000 11286.065838 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.534022] ptlrpcd_00_09 117143 116840710.217283 881411 120 0.000000 22636.572398 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.545451] ll_cfg_requeue 117287 100576837.842933 42 120 0.000000 1.063548 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.556878] mdt_attr00_001 117484 98721370.120677 3 120 0.000000 0.034816 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.568300] mdt_io00_001 117508 100577176.614349 151 120 0.000000 5.902065 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.579736] osp-syn-14-2 117580 116840703.774560 238075 120 0.000000 6373.466194 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.591187] lquota_wb_fir-M 117696 116840793.398082 3421 120 0.000000 64.048363 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.602608] qmt_reba_fir-QM 117710 107479905.708630 24 120 0.000000 27.194839 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.614060] mdt_rdpg00_003 118086 116840973.529897 18840888 120 0.000000 985283.529868 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.625483] mdt00_012 118160 116838333.322604 24078535 120 0.000000 1258451.802931 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.636903] mdt00_014 118168 99832048.774237 2196164 120 0.000000 76056.270149 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.648345] mdt00_030 118223 116834681.112121 29476632 120 0.000000 1787320.208709 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.659777] mdt00_040 118253 116839594.568010 28657262 120 0.000000 1671702.330049 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.671200] mdt00_048 118278 116840703.894377 34478467 120 0.000000 1916063.457924 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.682623] mdt00_058 118308 104662007.058264 5176559 120 0.000000 221888.146861 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.694477] mdt00_062 118361 107688095.091556 13166988 120 0.000000 669441.050505 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.705921] mdt_io00_004 118416 116840957.535258 45129 120 0.000000 3681.435535 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.717349] mdt00_074 118419 116834681.049423 27989644 120 0.000000 1595741.756124 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.728783] R mdt00_093 118536 116840995.137469 43444786 120 0.000000 2038619.972640 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.740204] mdt00_095 118547 116838162.797894 31089728 120 0.000000 1648851.614801 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.751648] mdt_io00_009 118607 116839594.709700 3337554 120 0.000000 221321.702231 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.763076] mdt_io00_014 118621 100577020.702096 136 120 0.000000 6.506174 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.774501] mdt_io00_019 118639 100577178.016680 235 120 0.000000 12.766812 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.785925] mdt_io00_024 118656 116839594.581486 23586 120 0.000000 2144.088531 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.797353] mdt_io00_036 118743 116839598.334564 2693479 120 0.000000 211802.428797 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.808779] mdt_io00_041 118780 100577020.330835 77 120 0.000000 4.333085 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.820208] mdt_io00_046 118800 116841007.791156 34431 120 0.000000 3086.203140 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.831661] mdt_io00_087 118983 116839595.200132 1773367 120 0.000000 146352.091486 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.843101] mdt_io00_097 119026 100577020.518231 729871 120 0.000000 29379.028128 0.000000 0 / [Fri Apr 12 11:14:00 2019][2937762.854547] mdt_io00_108 119127 100577020.907601 77 120 0.000000 2.799390 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.865976] mdt_rdpg00_028 119128 116840438.440761 21829951 120 0.000000 991511.146692 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.877972] mdt_rdpg00_040 119186 116840439.241332 23321414 120 0.000000 1077531.285236 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.889526] mdt_rdpg00_041 119192 116840438.442395 21235803 120 0.000000 997383.557657 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.900952] mdt_rdpg00_050 119228 116840624.341942 23269631 120 0.000000 1080285.710504 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.912376] mdt_rdpg00_056 119250 116841032.299798 16093652 120 0.000000 860954.872183 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.923814] ldlm_cn00_013 123639 116840962.971993 41790812 120 0.000000 1009839.361799 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.935245] ldlm_cb00_003 124118 106228015.073295 182 120 0.000000 5.572036 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.946669] ldlm_cn00_016 127039 116841023.472999 38067112 120 0.000000 929955.034314 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.958089] ldlm_bl_14 128947 116840965.543027 700725 120 0.000000 60701.747979 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.969512] ldlm_cn00_023 129947 116840962.532093 34785389 120 0.000000 862388.906465 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.980939] osp-pre-13-3 2235 116840533.662514 28033 120 0.000000 586.574415 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937762.992273] osp-pre-31-3 2249 116840876.183065 28022 120 0.000000 616.571410 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937763.003610] osp-pre-28-3 2323 116840514.645971 27856 120 0.000000 599.759241 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937763.014955] ldlm_cn00_031 6488 112594573.445032 29609138 120 0.000000 727286.220259 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937763.026293] kworker/0:0 24961 106779730.785914 2 120 0.000000 0.010641 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937763.037645] kworker/u257:2 34641 116840887.400392 306299 120 0.000000 50272.514143 0.000000 0 / [Fri Apr 12 11:14:01 2019][2937763.049000] [Fri Apr 12 11:14:01 2019][2937763.050680] cpu#1, 1996.203 MHz [Fri Apr 12 11:14:01 2019][2937763.054001] .nr_running : 2 [Fri Apr 12 11:14:01 2019][2937763.058804] .load : 1024 [Fri Apr 12 11:14:01 2019][2937763.063863] .nr_switches : 20207242426 [Fri Apr 12 11:14:01 2019][2937763.069523] .nr_load_updates : 2170996154 [Fri Apr 12 11:14:01 2019][2937763.075102] .nr_uninterruptible : -68660 [Fri Apr 12 11:14:01 2019][2937763.080329] .next_balance : 7232.363826 [Fri Apr 12 11:14:01 2019][2937763.085988] .curr->pid : 117122 [Fri Apr 12 11:14:01 2019][2937763.091215] .clock : 2937763091.211077 [Fri Apr 12 11:14:01 2019][2937763.097396] .cpu_load[0] : 1024 [Fri Apr 12 11:14:01 2019][2937763.102457] .cpu_load[1] : 574 [Fri Apr 12 11:14:01 2019][2937763.107421] .cpu_load[2] : 380 [Fri Apr 12 11:14:01 2019][2937763.112387] .cpu_load[3] : 193 [Fri Apr 12 11:14:01 2019][2937763.117375] .cpu_load[4] : 382 [Fri Apr 12 11:14:01 2019][2937763.122346] .avg_idle : 96100 [Fri Apr 12 11:14:01 2019][2937763.127487] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:01 2019][2937763.132807] [Fri Apr 12 11:14:01 2019][2937763.134476] cfs_rq[1]:/ [Fri Apr 12 11:14:01 2019][2937763.137102] .exec_clock : 0.000000 [Fri Apr 12 11:14:01 2019][2937763.142502] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:01 2019][2937763.147903] .min_vruntime : 129027730.692121 [Fri Apr 12 11:14:01 2019][2937763.153996] .max_vruntime : 0.000001 [Fri Apr 12 11:14:01 2019][2937763.159394] .spread : 0.000000 [Fri Apr 12 11:14:01 2019][2937763.164793] .spread0 : 12186671.802788 [Fri Apr 12 11:14:01 2019][2937763.170798] .nr_spread_over : 0 [Fri Apr 12 11:14:01 2019][2937763.175592] .nr_running : 1 [Fri Apr 12 11:14:01 2019][2937763.180384] .load : 0 [Fri Apr 12 11:14:01 2019][2937763.185177] .runnable_load_avg : 0 [Fri Apr 12 11:14:01 2019][2937763.189970] .blocked_load_avg : 1031 [Fri Apr 12 11:14:01 2019][2937763.195022] .tg_load_avg : 0 [Fri Apr 12 11:14:01 2019][2937763.199817] .tg_load_contrib : 0 [Fri Apr 12 11:14:01 2019][2937763.204609] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:01 2019][2937763.209402] .tg->runnable_avg : 0 [Fri Apr 12 11:14:01 2019][2937763.214193] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:01 2019][2937763.218989] .throttled : 0 [Fri Apr 12 11:14:01 2019][2937763.223781] .throttle_count : 0 [Fri Apr 12 11:14:01 2019][2937763.228572] .avg->runnable_avg_sum : 24159 [Fri Apr 12 11:14:01 2019][2937763.233714] .avg->runnable_avg_period : 46789 [Fri Apr 12 11:14:01 2019][2937763.238859] [Fri Apr 12 11:14:01 2019][2937763.240535] rt_rq[1]:/ [Fri Apr 12 11:14:01 2019][2937763.243073] .rt_nr_running : 0 [Fri Apr 12 11:14:01 2019][2937763.247866] .rt_throttled : 0 [Fri Apr 12 11:14:01 2019][2937763.252658] .rt_time : 0.008566 [Fri Apr 12 11:14:01 2019][2937763.258059] .rt_runtime : 837.266119 [Fri Apr 12 11:14:01 2019][2937763.263630] [Fri Apr 12 11:14:01 2019][2937763.265301] runnable tasks: [Fri Apr 12 11:14:01 2019][2937763.268277] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:01 2019][2937763.279241] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:01 2019][2937763.290110] watchdog/1 13 -2.968009 734427 0 0.000000 52009.877080 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.301442] migration/1 14 0.000000 47662 0 0.000000 615587.321667 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.312780] ksoftirqd/1 15 129027748.005232 37857692 120 0.000000 573657.151191 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.324115] kworker/1:0H 17 4337.247943 9 100 0.000000 0.030096 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.335491] kworker/u263:0 7396 2179.832998 4 100 0.000000 0.015861 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.346870] kworker/1:1H 17603 129027747.988704 30815817 100 0.000000 202189.579528 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.358222] dsm_sa_datamgrd 18871 5729.373237 1694 120 0.000000 1302.928613 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.369572] agetty 19555 5798.321771 13 120 0.000000 10.992759 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.380959] kworker/1:1 96840 129027755.232608 446030 120 0.000000 10161.071520 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.392312] ptlrpcd_01_10 117156 129026669.888427 650380 120 0.000000 16363.274686 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.403747] mdt01_000 117466 128934040.290797 23263454 120 0.000000 1576959.150441 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.415178] mdt01_001 117467 118241723.254946 5928985 120 0.000000 291150.215118 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.426607] mdt_attr01_000 117485 111004137.576093 3 120 0.000000 0.081544 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.438042] mdt_attr01_001 117486 111004137.595340 3 120 0.000000 0.028114 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.449525] mdt01_009 118148 112865268.036244 5037407 120 0.000000 189781.843295 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.460954] mdt01_021 118185 128961608.879654 24639733 120 0.000000 1751822.588944 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.472382] mdt01_048 118275 113125289.372002 5106140 120 0.000000 195538.518570 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.483808] R mdt01_064 118323 129027832.323105 21795343 120 0.000000 1608102.326314 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.495227] mdt01_067 118327 128929943.240739 25097985 120 0.000000 1691101.946797 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.506649] mdt01_068 118329 128962192.153808 30185865 120 0.000000 1977274.811729 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.518076] mdt01_082 118349 129027762.000492 26148909 120 0.000000 1793537.219422 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.529498] mdt01_098 118389 129027867.867295 26659301 120 0.000000 1811510.831881 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.540918] mdt01_103 118403 129016446.463006 28899017 120 0.000000 1894850.178147 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.552371] mdt_io01_022 118737 119502308.008914 15387 120 0.000000 707.991243 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.563798] mdt_io01_035 118827 128379615.418551 3690102 120 0.000000 233502.259631 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.575260] mdt_io01_083 119075 113405031.839792 67 120 0.000000 3.555228 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.586680] mdt_io01_085 119093 113405035.597018 58 120 0.000000 2.408255 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.598108] mdt_io01_092 119122 119502308.013743 1617 120 0.000000 81.691109 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.609536] mdt_io01_098 119150 119502308.009335 362678 120 0.000000 16303.169029 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.620961] mdt_io01_103 119188 119502308.012611 528364 120 0.000000 18275.707383 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.632381] mdt_io01_105 119200 129019848.584114 1825272 120 0.000000 171446.172985 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.643802] mdt_rdpg01_048 119201 129027733.658435 30410481 120 0.000000 1388805.027574 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.655238] ldlm_cb01_002 122132 119200906.603170 713 120 0.000000 21.859310 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.666695] kworker/1:0 25476 119602793.634841 2 120 0.000000 0.008996 0.000000 1 / [Fri Apr 12 11:14:01 2019][2937763.678049] [Fri Apr 12 11:14:01 2019][2937763.679720] cpu#2, 1996.203 MHz [Fri Apr 12 11:14:01 2019][2937763.683053] .nr_running : 1 [Fri Apr 12 11:14:01 2019][2937763.687863] .load : 0 [Fri Apr 12 11:14:01 2019][2937763.692663] .nr_switches : 10874764960 [Fri Apr 12 11:14:01 2019][2937763.698334] .nr_load_updates : 1892103575 [Fri Apr 12 11:14:01 2019][2937763.703912] .nr_uninterruptible : -69134 [Fri Apr 12 11:14:01 2019][2937763.709148] .next_balance : 7232.364454 [Fri Apr 12 11:14:01 2019][2937763.714817] .curr->pid : 119056 [Fri Apr 12 11:14:01 2019][2937763.720070] .clock : 2937763719.915502 [Fri Apr 12 11:14:01 2019][2937763.726273] .cpu_load[0] : 1024 [Fri Apr 12 11:14:01 2019][2937763.731339] .cpu_load[1] : 78 [Fri Apr 12 11:14:01 2019][2937763.736222] .cpu_load[2] : 130 [Fri Apr 12 11:14:01 2019][2937763.741199] .cpu_load[3] : 640 [Fri Apr 12 11:14:01 2019][2937763.746180] .cpu_load[4] : 1409 [Fri Apr 12 11:14:01 2019][2937763.751253] .avg_idle : 34576 [Fri Apr 12 11:14:01 2019][2937763.756400] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:01 2019][2937763.761739] [Fri Apr 12 11:14:01 2019][2937763.763416] cfs_rq[2]:/ [Fri Apr 12 11:14:01 2019][2937763.766041] .exec_clock : 0.000000 [Fri Apr 12 11:14:01 2019][2937763.771440] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:01 2019][2937763.776840] .min_vruntime : 87317782.020672 [Fri Apr 12 11:14:01 2019][2937763.782846] .max_vruntime : 0.000001 [Fri Apr 12 11:14:01 2019][2937763.788246] .spread : 0.000000 [Fri Apr 12 11:14:01 2019][2937763.793645] .spread0 : -29523295.032054 [Fri Apr 12 11:14:01 2019][2937763.799737] .nr_spread_over : 0 [Fri Apr 12 11:14:01 2019][2937763.804530] .nr_running : 0 [Fri Apr 12 11:14:01 2019][2937763.809322] .load : 1024 [Fri Apr 12 11:14:01 2019][2937763.814374] .runnable_load_avg : 0 [Fri Apr 12 11:14:01 2019][2937763.819169] .blocked_load_avg : 823 [Fri Apr 12 11:14:01 2019][2937763.824134] .tg_load_avg : 0 [Fri Apr 12 11:14:01 2019][2937763.828928] .tg_load_contrib : 0 [Fri Apr 12 11:14:01 2019][2937763.833719] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:01 2019][2937763.838512] .tg->runnable_avg : 0 [Fri Apr 12 11:14:01 2019][2937763.843304] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:01 2019][2937763.848099] .throttled : 0 [Fri Apr 12 11:14:01 2019][2937763.852891] .throttle_count : 0 [Fri Apr 12 11:14:01 2019][2937763.857683] .avg->runnable_avg_sum : 25403 [Fri Apr 12 11:14:01 2019][2937763.862824] .avg->runnable_avg_period : 46983 [Fri Apr 12 11:14:02 2019][2937763.867970] [Fri Apr 12 11:14:02 2019][2937763.869643] rt_rq[2]:/ [Fri Apr 12 11:14:02 2019][2937763.872182] .rt_nr_running : 0 [Fri Apr 12 11:14:02 2019][2937763.876974] .rt_throttled : 0 [Fri Apr 12 11:14:02 2019][2937763.881770] .rt_time : 0.000000 [Fri Apr 12 11:14:02 2019][2937763.887168] .rt_runtime : 875.467908 [Fri Apr 12 11:14:02 2019][2937763.892740] [Fri Apr 12 11:14:02 2019][2937763.894412] runnable tasks: [Fri Apr 12 11:14:02 2019][2937763.897396] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:02 2019][2937763.908349] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:02 2019][2937763.919212] watchdog/2 19 -5.961086 734427 0 0.000000 25522.274050 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937763.930544] migration/2 20 0.000000 29459 0 0.000000 343355.719115 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937763.941880] ksoftirqd/2 21 87317763.696142 31754174 120 0.000000 425753.931358 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937763.953218] kworker/2:0H 23 5114.763384 10 100 0.000000 0.037061 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937763.964583] irq/28-AMD-Vi 271 0.000000 3 49 0.000000 0.030678 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937763.975918] kswapd2 279 292.530163 4 120 0.000000 0.039314 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937763.987282] polkitd 18057 4997.809931 3 120 0.000000 0.069051 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937763.998618] sssd_be 18103 87316998.545462 2003917 120 0.000000 1904255.118955 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.009960] kworker/2:1H 19433 87317839.372557 25693337 100 0.000000 145300.004329 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.021310] kworker/2:0 103156 87317692.539564 867133 120 0.000000 15259.604762 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.032741] kworker/2:1 21724 68748930.619633 3 120 0.000000 0.013906 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.044084] ptlrpcd_02_01 117159 87317213.723086 855904 120 0.000000 21494.974718 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.055511] ll_mgs_0004 117300 87317448.682642 4296037 120 0.000000 137566.295885 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.066937] mdt02_001 117470 73259224.188725 199256 120 0.000000 11359.328745 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.078359] mdt02_002 117471 87293708.417853 14067599 120 0.000000 944576.829151 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.089784] mdt_attr02_000 117487 71300393.158468 3 120 0.000000 0.063641 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.101205] mdt_attr02_001 117488 71300393.177204 3 120 0.000000 0.028184 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.112634] mdt_io02_001 117514 73191953.243136 124 120 0.000000 6.019550 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.124061] mdt02_004 117532 87237088.691869 16457747 120 0.000000 1146508.476905 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.135485] mdt02_008 117558 87290784.993048 16429675 120 0.000000 1161810.875751 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.146964] mdt02_025 118245 87226162.672946 14903191 120 0.000000 1103295.462965 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.158401] mdt02_051 118397 87206625.460542 13253586 120 0.000000 981268.020548 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.169823] mdt02_057 118442 87310075.504739 22544547 120 0.000000 1450563.857437 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.181243] mdt02_065 118472 71566938.666612 1811003 120 0.000000 64029.855267 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.192667] mdt02_071 118484 87276583.263303 20086420 120 0.000000 1483807.316656 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.204093] mdt02_091 118532 83564362.990112 16921889 120 0.000000 1145353.837814 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.215516] mdt02_101 118577 72169933.264635 3192027 120 0.000000 126144.991100 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.226935] mdt02_107 118585 87212156.792685 27365876 120 0.000000 1568228.700015 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.238367] mdt_io02_024 118670 73191953.401208 77 120 0.000000 4.172848 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.249793] mdt_io02_036 118707 73192374.792543 856659 120 0.000000 35938.651483 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.261216] mdt_io02_050 118760 73191953.379506 32 120 0.000000 1.278220 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.272638] mdt_io02_054 118785 73191953.390044 52 120 0.000000 2.662762 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.284059] mdt_io02_061 118807 73191956.682241 111 120 0.000000 5.883303 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.295480] mdt_io02_062 118809 73192374.620187 87 120 0.000000 3.808969 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.306904] mdt_io02_063 118811 86595985.076096 3083146 120 0.000000 252322.960674 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.318329] mdt_io02_066 118820 73191956.783704 823179 120 0.000000 33722.212391 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.329756] mdt_io02_073 118859 73191956.238292 42 120 0.000000 2.077524 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.341185] mdt_io02_077 118883 73191956.155176 60 120 0.000000 2.720207 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.352607] mdt_io02_085 118905 73192375.130477 52 120 0.000000 2.637323 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.364037] mdt_io02_102 118968 73191956.762484 639902 120 0.000000 26991.717550 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.375465] mdt_rdpg02_007 119019 87283291.808802 16452853 120 0.000000 890534.550816 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.386894] mdt_rdpg02_010 119034 87283286.481252 21466505 120 0.000000 1023143.858004 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.398321] mdt_rdpg02_020 119080 87317879.782330 20170560 120 0.000000 1110939.849071 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.409745] mdt_rdpg02_021 119081 87313200.672319 26668242 120 0.000000 1335192.839871 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.421175] mdt_rdpg02_030 119123 87283291.848508 18902509 120 0.000000 1005959.168586 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.432601] mdt_rdpg02_033 119139 87283291.808211 22033790 120 0.000000 1079452.367078 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.444029] mdt_rdpg02_045 119185 87305604.994884 15521208 120 0.000000 893446.155824 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.455457] mdt_rdpg02_049 119199 87283286.481694 17548234 120 0.000000 929618.659818 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.466879] mdt_rdpg02_050 119211 87283286.481112 19752139 120 0.000000 963250.590478 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.478305] mdt_rdpg02_058 119246 87283286.481073 14964812 120 0.000000 712101.208240 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.489789] kworker/u259:2 46727 87317768.874668 41 120 0.000000 8.977195 0.000000 2 / [Fri Apr 12 11:14:02 2019][2937764.501142] [Fri Apr 12 11:14:02 2019][2937764.502823] cpu#3, 1996.203 MHz [Fri Apr 12 11:14:02 2019][2937764.506151] .nr_running : 0 [Fri Apr 12 11:14:02 2019][2937764.510953] .load : 0 [Fri Apr 12 11:14:02 2019][2937764.515752] .nr_switches : 9894510161 [Fri Apr 12 11:14:02 2019][2937764.521326] .nr_load_updates : 1792262768 [Fri Apr 12 11:14:02 2019][2937764.526899] .nr_uninterruptible : -32336 [Fri Apr 12 11:14:02 2019][2937764.532126] .next_balance : 7232.365278 [Fri Apr 12 11:14:02 2019][2937764.537786] .curr->pid : 0 [Fri Apr 12 11:14:02 2019][2937764.542586] .clock : 2937764542.574329 [Fri Apr 12 11:14:02 2019][2937764.548765] .cpu_load[0] : 0 [Fri Apr 12 11:14:02 2019][2937764.553558] .cpu_load[1] : 5 [Fri Apr 12 11:14:02 2019][2937764.558351] .cpu_load[2] : 93 [Fri Apr 12 11:14:02 2019][2937764.563231] .cpu_load[3] : 70 [Fri Apr 12 11:14:02 2019][2937764.568109] .cpu_load[4] : 64 [Fri Apr 12 11:14:02 2019][2937764.572989] .avg_idle : 51555 [Fri Apr 12 11:14:02 2019][2937764.578128] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:02 2019][2937764.583440] [Fri Apr 12 11:14:02 2019][2937764.585112] cfs_rq[3]:/ [Fri Apr 12 11:14:02 2019][2937764.587737] .exec_clock : 0.000000 [Fri Apr 12 11:14:02 2019][2937764.593136] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:02 2019][2937764.598535] .min_vruntime : 73300590.145970 [Fri Apr 12 11:14:02 2019][2937764.604542] .max_vruntime : 0.000001 [Fri Apr 12 11:14:02 2019][2937764.609942] .spread : 0.000000 [Fri Apr 12 11:14:02 2019][2937764.615340] .spread0 : -43540670.367805 [Fri Apr 12 11:14:02 2019][2937764.621433] .nr_spread_over : 0 [Fri Apr 12 11:14:02 2019][2937764.626225] .nr_running : 0 [Fri Apr 12 11:14:02 2019][2937764.631019] .load : 0 [Fri Apr 12 11:14:02 2019][2937764.635812] .runnable_load_avg : 0 [Fri Apr 12 11:14:02 2019][2937764.640605] .blocked_load_avg : 110 [Fri Apr 12 11:14:02 2019][2937764.645572] .tg_load_avg : 0 [Fri Apr 12 11:14:02 2019][2937764.650363] .tg_load_contrib : 0 [Fri Apr 12 11:14:02 2019][2937764.655157] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:02 2019][2937764.659950] .tg->runnable_avg : 0 [Fri Apr 12 11:14:02 2019][2937764.664742] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:02 2019][2937764.669534] .throttled : 0 [Fri Apr 12 11:14:02 2019][2937764.674327] .throttle_count : 0 [Fri Apr 12 11:14:02 2019][2937764.679121] .avg->runnable_avg_sum : 6401 [Fri Apr 12 11:14:02 2019][2937764.684172] .avg->runnable_avg_period : 47053 [Fri Apr 12 11:14:02 2019][2937764.689319] [Fri Apr 12 11:14:02 2019][2937764.690995] rt_rq[3]:/ [Fri Apr 12 11:14:02 2019][2937764.693534] .rt_nr_running : 0 [Fri Apr 12 11:14:02 2019][2937764.698325] .rt_throttled : 0 [Fri Apr 12 11:14:02 2019][2937764.703119] .rt_time : 0.000000 [Fri Apr 12 11:14:02 2019][2937764.708518] .rt_runtime : 950.000000 [Fri Apr 12 11:14:02 2019][2937764.714091] [Fri Apr 12 11:14:02 2019][2937764.715764] runnable tasks: [Fri Apr 12 11:14:02 2019][2937764.718735] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:02 2019][2937764.729681] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:02 2019][2937764.740546] watchdog/3 25 -5.963530 734427 0 0.000000 25757.197065 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.751879] migration/3 26 0.000000 8805 0 0.000000 98880.201412 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.763212] ksoftirqd/3 27 73300595.309025 19907548 120 0.000000 205462.060729 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.774552] kworker/3:0H 29 4341.581372 15 100 0.000000 0.071685 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.785916] irq/29-AMD-Vi 272 0.000000 3 49 0.000000 0.035928 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.797249] kswapd3 280 -1.009863 4 120 0.000000 0.037320 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.808585] khugepaged 282 73300136.671805 510924 139 0.000000 1887211.640293 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.819946] kdmwork-253:0 17105 73300603.003790 265933367 120 0.000000 4118181.370248 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.831287] auditd 17979 73275946.167152 23854 116 0.000000 691.724365 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.842634] slapd 19691 4348.258083 1808 120 0.000000 217.329790 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.853967] kworker/3:1H 20552 73300606.459907 17756472 100 0.000000 90185.060605 0.000000 3 / [Fri Apr 12 11:14:02 2019][2937764.865310] kworker/3:1 51272 73300556.529728 552185 120 0.000000 9543.344374 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.876652] cfs_rh_02 117054 73300429.126464 9842 120 0.000000 6393.480113 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.888078] ptlrpc_hr03_000 117093 73300567.097475 1488566 120 0.000000 28002.714077 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.899504] kiblnd_sd_03_01 117128 71834164.118425 1410919867 120 0.000000 11221622.452324 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.911012] lnet_discovery 117132 73300610.027973 189570 120 0.000000 1513.785514 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.922449] mdt_rdpg03_001 117482 73286900.924077 17123015 120 0.000000 785092.321266 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.933875] mdt_attr03_000 117489 53911419.418542 3 120 0.000000 0.082136 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.945296] mdt_attr03_001 117490 53911419.437598 3 120 0.000000 0.028775 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.956724] mdt_io03_002 117518 56392237.438592 53 120 0.000000 2.212369 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.968145] mdt03_004 117548 73252529.147340 7590915 120 0.000000 335422.410329 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.979567] osp-pre-3-2 117567 73300327.598777 36802 120 0.000000 608.895312 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937764.990989] osp-syn-39-2 117572 73300612.208695 236027 120 0.000000 6405.864372 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.002414] osp-syn-19-2 117594 73300612.413106 238126 120 0.000000 6697.496219 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.013834] osp-syn-18-2 117596 73300612.048980 238743 120 0.000000 6350.950133 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.025259] osp-pre-25-2 117599 73300491.683487 40260 120 0.000000 996.813922 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.036683] osp-syn-33-2 117620 73300611.711347 237076 120 0.000000 6454.021158 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.048104] osp-pre-7-2 117625 73300539.168489 39973 120 0.000000 936.185011 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.059529] osp-pre-22-2 117645 73300495.866748 40342 120 0.000000 864.910799 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.070947] osp-syn-3-2 117646 73300612.048720 236563 120 0.000000 6266.666575 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.082374] osp-pre-37-2 117651 73300536.203515 42528 120 0.000000 938.114993 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.093804] osp-pre-3-0 117719 73300325.342910 36801 120 0.000000 617.035438 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.105228] osp-syn-46-0 117726 73258782.699025 75146 120 0.000000 2293.254055 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.116649] osp-syn-11-0 117730 73259055.552953 75001 120 0.000000 2287.119549 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.128074] osp-pre-2-0 117743 73300455.836583 39254 120 0.000000 846.045493 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.139496] osp-syn-19-0 117746 73288204.340302 75702 120 0.000000 2317.245246 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.150920] osp-syn-18-0 117748 73288440.868334 75928 120 0.000000 2192.524361 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.162351] osp-pre-25-0 117751 73300494.032299 39345 120 0.000000 843.930165 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.173783] osp-pre-44-0 117759 73300394.147867 39891 120 0.000000 833.364716 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.185233] osp-syn-16-0 117764 73288662.231321 75235 120 0.000000 2200.930608 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.196681] osp-pre-4-0 117769 73300364.808753 40590 120 0.000000 856.602376 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.208110] osp-syn-33-0 117772 73288651.619068 75508 120 0.000000 2297.678721 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.219551] osp-syn-40-0 117778 73288558.963348 74008 120 0.000000 2438.259779 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.230989] dist_txn-0 117837 73261581.301781 474 120 0.000000 33.932158 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.242417] mdt_rdpg03_002 118082 73300616.774442 11925693 120 0.000000 529093.092762 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.253875] mdt_rdpg03_005 118491 73300643.841019 14524682 120 0.000000 635730.712289 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.265301] mdt_rdpg03_007 118496 73300453.552722 12397065 120 0.000000 504070.255070 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.276726] mdt_rdpg03_010 118505 73300402.118030 15685689 120 0.000000 688918.323275 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.288148] mdt_rdpg03_011 118506 73294327.838985 14484269 120 0.000000 603258.614201 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.299575] mdt_rdpg03_022 118540 73286900.926171 13512986 120 0.000000 545624.188965 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.311001] mdt_io03_009 118625 56392236.490237 18 120 0.000000 1.384558 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.322430] mdt_io03_011 118640 56392237.447638 169 120 0.000000 4.411835 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.333850] mdt_io03_013 118649 72846143.753124 883793 120 0.000000 38356.417359 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.345275] mdt_io03_017 118668 56392236.486378 52 120 0.000000 2.476479 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.356704] mdt_io03_038 118766 56392236.430083 37 120 0.000000 1.684559 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.368127] mdt_io03_045 118793 72846143.752012 1214171 120 0.000000 94521.734053 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.379550] mdt_io03_047 118802 56392237.532911 42 120 0.000000 2.234252 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.390980] mdt_io03_059 118841 73274559.869840 959368 120 0.000000 42710.558246 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.402405] mdt_io03_061 118844 73298788.993411 3168783 120 0.000000 199217.953186 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.413826] mdt_io03_062 118846 73300661.308435 2138182 120 0.000000 158472.547856 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.425253] mdt_io03_068 118869 72846317.730279 859567 120 0.000000 44798.932119 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.436673] mdt_io03_070 118875 56392238.304839 52 120 0.000000 2.219112 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.448100] mdt_io03_075 118895 56392237.435787 39 120 0.000000 1.429197 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.459557] mdt_rdpg03_032 119134 73286900.928987 17937807 120 0.000000 658939.763573 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.470979] mdt_rdpg03_033 119151 73297721.382471 17062056 120 0.000000 792423.137724 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.482407] mdt_rdpg03_038 119212 73300181.448229 16710889 120 0.000000 613368.687167 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.493838] mdt_rdpg03_047 119253 73294327.838775 16022161 120 0.000000 645590.215100 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.505264] mdt_fld_0003 119284 72418919.796392 829 120 0.000000 13.027290 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.516691] ldlm_cb03_003 122541 58677073.683582 289 120 0.000000 6.728440 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.528129] mdt_out03_003 1712 73300660.526997 135459 120 0.000000 4615.484467 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.539462] osp-pre-2-3 2224 73300574.390347 27362 120 0.000000 483.553328 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.550801] osp-syn-10-3 2240 73300417.013690 9575 120 0.000000 283.684952 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.562136] osp-syn-43-3 2256 73300417.031155 9571 120 0.000000 289.627582 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.573471] osp-syn-19-3 2258 73300394.180629 9546 120 0.000000 288.912544 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.584808] osp-pre-18-3 2261 73300556.261660 27872 120 0.000000 503.560939 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.596142] osp-syn-17-3 2264 73300494.046455 9470 120 0.000000 287.353070 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.607489] osp-pre-16-3 2277 73300364.639768 28018 120 0.000000 511.502375 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.618842] osp-syn-4-3 2280 73299833.216776 9509 120 0.000000 289.769655 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.630177] osp-syn-23-3 2282 73300416.934470 9454 120 0.000000 281.600631 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.641525] osp-syn-45-3 2292 73300556.668006 9419 120 0.000000 315.119008 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.652894] osp-pre-45-3 2293 73300556.654410 27852 120 0.000000 653.280686 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.664231] osp-syn-6-3 2294 73300416.925636 9523 120 0.000000 274.924107 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.675577] osp-syn-3-3 2310 73300294.394308 9441 120 0.000000 294.452616 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.686913] osp-syn-8-3 2318 73299750.527422 9383 120 0.000000 318.257410 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.698257] osp-syn-35-3 2320 73300109.385086 9644 120 0.000000 327.231995 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.709593] osp-syn-30-3 2324 73299965.898888 9462 120 0.000000 302.335541 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.720939] osp_up3-1 2388 73267043.761380 405 120 0.000000 24.877660 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.732285] osp-pre-10-1 2399 73300527.357095 27713 120 0.000000 543.565210 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.743620] osp-pre-11-1 2401 73300463.416092 27745 120 0.000000 665.775759 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.754956] osp-pre-43-1 2415 73300553.975927 27755 120 0.000000 607.653310 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.766292] osp-syn-32-1 2430 73298393.204090 19687 120 0.000000 485.703333 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.777628] osp-syn-33-1 2442 73298518.432351 19993 120 0.000000 467.602507 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.788965] osp-pre-24-1 2455 73300402.129522 28037 120 0.000000 545.640429 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.800300] osp-pre-47-1 2457 73300468.027575 27625 120 0.000000 651.568299 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.811647] osp-pre-15-1 2461 73300390.337573 27563 120 0.000000 801.024200 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.822983] osp-pre-29-1 2463 73300375.039344 27839 120 0.000000 613.639794 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.834328] osp-pre-1-1 2465 73300572.835163 28049 120 0.000000 671.680366 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.845670] osp-pre-22-1 2467 73300415.995976 27727 120 0.000000 599.827784 0.000000 3 / [Fri Apr 12 11:14:03 2019][2937765.857037] mdt_io03_098 3070 73294188.475915 497164 120 0.000000 37560.994591 0.000000 3 / [Fri Apr 12 11:14:04 2019][2937765.868381] mdt_io03_099 3072 56392248.234520 5 120 0.000000 0.140388 0.000000 3 / [Fri Apr 12 11:14:04 2019][2937765.879722] kworker/3:0 27935 62264303.439681 2 120 0.000000 0.009438 0.000000 3 / [Fri Apr 12 11:14:04 2019][2937765.891123] [Fri Apr 12 11:14:04 2019][2937765.892798] cpu#4, 1996.203 MHz [Fri Apr 12 11:14:04 2019][2937765.896127] .nr_running : 0 [Fri Apr 12 11:14:04 2019][2937765.900938] .load : 0 [Fri Apr 12 11:14:04 2019][2937765.905749] .nr_switches : 15281141589 [Fri Apr 12 11:14:04 2019][2937765.911423] .nr_load_updates : 2118737917 [Fri Apr 12 11:14:04 2019][2937765.917025] .nr_uninterruptible : 712885 [Fri Apr 12 11:14:04 2019][2937765.922259] .next_balance : 7232.366669 [Fri Apr 12 11:14:04 2019][2937765.927947] .curr->pid : 0 [Fri Apr 12 11:14:04 2019][2937765.932763] .clock : 2937765932.372451 [Fri Apr 12 11:14:04 2019][2937765.938947] .cpu_load[0] : 0 [Fri Apr 12 11:14:04 2019][2937765.943751] .cpu_load[1] : 216 [Fri Apr 12 11:14:04 2019][2937765.948729] .cpu_load[2] : 282 [Fri Apr 12 11:14:04 2019][2937765.953698] .cpu_load[3] : 183 [Fri Apr 12 11:14:04 2019][2937765.958665] .cpu_load[4] : 222 [Fri Apr 12 11:14:04 2019][2937765.963632] .avg_idle : 68015 [Fri Apr 12 11:14:04 2019][2937765.968770] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:04 2019][2937765.974083] [Fri Apr 12 11:14:04 2019][2937765.975753] cfs_rq[4]:/ [Fri Apr 12 11:14:04 2019][2937765.978381] .exec_clock : 0.000000 [Fri Apr 12 11:14:04 2019][2937765.983780] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:04 2019][2937765.989179] .min_vruntime : 109812860.214478 [Fri Apr 12 11:14:04 2019][2937765.995276] .max_vruntime : 0.000001 [Fri Apr 12 11:14:04 2019][2937766.000678] .spread : 0.000000 [Fri Apr 12 11:14:04 2019][2937766.006078] .spread0 : -7028619.679053 [Fri Apr 12 11:14:04 2019][2937766.012103] .nr_spread_over : 0 [Fri Apr 12 11:14:04 2019][2937766.016896] .nr_running : 0 [Fri Apr 12 11:14:04 2019][2937766.021688] .load : 0 [Fri Apr 12 11:14:04 2019][2937766.026488] .runnable_load_avg : 0 [Fri Apr 12 11:14:04 2019][2937766.031283] .blocked_load_avg : 215 [Fri Apr 12 11:14:04 2019][2937766.036247] .tg_load_avg : 0 [Fri Apr 12 11:14:04 2019][2937766.041041] .tg_load_contrib : 0 [Fri Apr 12 11:14:04 2019][2937766.045842] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:04 2019][2937766.050636] .tg->runnable_avg : 0 [Fri Apr 12 11:14:04 2019][2937766.055428] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:04 2019][2937766.060220] .throttled : 0 [Fri Apr 12 11:14:04 2019][2937766.065013] .throttle_count : 0 [Fri Apr 12 11:14:04 2019][2937766.069805] .avg->runnable_avg_sum : 11802 [Fri Apr 12 11:14:04 2019][2937766.074944] .avg->runnable_avg_period : 46297 [Fri Apr 12 11:14:04 2019][2937766.080093] [Fri Apr 12 11:14:04 2019][2937766.081765] rt_rq[4]:/ [Fri Apr 12 11:14:04 2019][2937766.084304] .rt_nr_running : 0 [Fri Apr 12 11:14:04 2019][2937766.089098] .rt_throttled : 0 [Fri Apr 12 11:14:04 2019][2937766.093889] .rt_time : 0.000000 [Fri Apr 12 11:14:04 2019][2937766.099291] .rt_runtime : 950.000000 [Fri Apr 12 11:14:04 2019][2937766.104862] [Fri Apr 12 11:14:04 2019][2937766.106534] runnable tasks: [Fri Apr 12 11:14:04 2019][2937766.109506] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:04 2019][2937766.120454] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:04 2019][2937766.131318] watchdog/4 31 -8.962809 734427 0 0.000000 38499.970658 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.142650] migration/4 32 0.000000 27637 0 0.000000 388689.440351 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.153987] ksoftirqd/4 33 109812874.630522 31782999 120 0.000000 428532.117311 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.165323] kworker/4:0H 35 2823.040273 9 100 0.000000 0.032569 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.176712] kworker/4:1H 16844 109812883.513137 29264704 100 0.000000 186308.411987 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.188075] kworker/4:2 66614 109812869.454291 558612 120 0.000000 11814.868380 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.199414] ptlrpc_hr00_004 117077 109812881.541822 2394922 120 0.000000 68859.580849 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.210854] kiblnd_sd_00_00 117115 109812903.485459 2266791974 120 0.000000 20001748.218295 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.222360] kiblnd_sd_00_02 117117 99751736.386649 2267513252 120 0.000000 19940575.280499 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.233871] ptlrpcd_00_02 117136 109812702.988544 855980 120 0.000000 23378.772442 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.245296] ldlm_cb00_001 117258 101403643.239382 361 120 0.000000 17.687705 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.256723] mdt_out00_000 117492 109718117.200237 12520240 120 0.000000 311258.660584 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.268192] osp-syn-28-2 117658 109812689.514295 236038 120 0.000000 6906.435793 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.279637] mdt00_007 117869 109749212.629958 28892095 120 0.000000 1548513.968995 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.291077] mdt00_013 118167 93436192.142718 3720018 120 0.000000 140175.722819 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.302512] mdt00_017 118177 98767654.481302 764971 120 0.000000 43531.865513 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.313944] mdt00_022 118200 109812387.192430 12850477 120 0.000000 1061776.378390 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.325373] mdt00_041 118257 109730571.119957 20880635 120 0.000000 1350352.087283 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.336846] mdt00_072 118409 93930654.988516 2443013 120 0.000000 94778.694698 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.348281] mdt_io00_004 118416 109812842.252015 45915 120 0.000000 3746.926687 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.359723] mdt00_081 118441 109812859.636848 31211258 120 0.000000 1524834.465600 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.371145] mdt00_090 118478 109812387.191908 25209058 120 0.000000 1383052.014149 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.382575] mdt_rdpg00_007 118495 109804637.658446 28245096 120 0.000000 1185527.073152 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.393999] mdt_rdpg00_009 118501 109809590.682171 20477156 120 0.000000 967218.135113 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.405425] mdt00_103 118558 109812389.590091 33876935 120 0.000000 1649380.999215 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.416864] mdt_io00_023 118653 93857362.905813 49 120 0.000000 2.266953 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.428303] mdt_io00_027 118673 93857143.788951 528746 120 0.000000 19181.207303 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.439723] mdt_io00_029 118685 93857374.130334 99 120 0.000000 4.789546 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.451153] mdt_io00_033 118726 93857145.554720 76 120 0.000000 4.276399 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.462594] mdt_io00_043 118788 109771530.403997 848329 120 0.000000 68621.700083 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.474029] mdt_io00_045 118795 93857362.748384 65 120 0.000000 3.318729 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.485467] mdt_io00_052 118834 93857143.761248 63 120 0.000000 3.852210 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.496897] mdt_io00_058 118861 109771530.357531 1534457 120 0.000000 136218.407312 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.508339] mdt_io00_077 118953 93857145.672134 69 120 0.000000 2.740341 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.519780] mdt_io00_104 119063 93857145.524693 40 120 0.000000 3.004887 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.531210] mdt_rdpg00_023 119101 109804637.658095 22610662 120 0.000000 1129977.630091 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.542799] mdt_rdpg00_024 119104 109809590.682922 22162620 120 0.000000 1029900.033443 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.554262] mdt_out00_002 119947 109718117.200208 14217997 120 0.000000 367149.223148 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.565692] kworker/4:1 122919 106243148.588577 9 120 0.000000 0.016710 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.577117] mdt_out00_003 126773 109718117.310256 15604239 120 0.000000 402531.037509 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.588561] osp-syn-46-3 2238 109808722.828353 9545 120 0.000000 303.157193 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.599922] osp-pre-36-3 2317 109812311.833399 28032 120 0.000000 580.683746 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.611258] osp-pre-34-3 2327 109812646.859587 27748 120 0.000000 736.579656 0.000000 0 / [Fri Apr 12 11:14:04 2019][2937766.622671] [Fri Apr 12 11:14:04 2019][2937766.624349] cpu#5, 1996.203 MHz [Fri Apr 12 11:14:04 2019][2937766.627687] .nr_running : 1 [Fri Apr 12 11:14:04 2019][2937766.632479] .load : 1024 [Fri Apr 12 11:14:04 2019][2937766.637533] .nr_switches : 18218713015 [Fri Apr 12 11:14:04 2019][2937766.643199] .nr_load_updates : 2157152180 [Fri Apr 12 11:14:04 2019][2937766.648772] .nr_uninterruptible : -82155 [Fri Apr 12 11:14:04 2019][2937766.654014] .next_balance : 7232.367400 [Fri Apr 12 11:14:04 2019][2937766.659677] .curr->pid : 117687 [Fri Apr 12 11:14:04 2019][2937766.664911] .clock : 2937766664.884485 [Fri Apr 12 11:14:04 2019][2937766.671089] .cpu_load[0] : 0 [Fri Apr 12 11:14:04 2019][2937766.675883] .cpu_load[1] : 262 [Fri Apr 12 11:14:04 2019][2937766.680849] .cpu_load[2] : 252 [Fri Apr 12 11:14:04 2019][2937766.685814] .cpu_load[3] : 1136 [Fri Apr 12 11:14:04 2019][2937766.690868] .cpu_load[4] : 746 [Fri Apr 12 11:14:04 2019][2937766.695833] .avg_idle : 155920 [Fri Apr 12 11:14:04 2019][2937766.701060] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:04 2019][2937766.706374] [Fri Apr 12 11:14:04 2019][2937766.708051] cfs_rq[5]:/ [Fri Apr 12 11:14:04 2019][2937766.710679] .exec_clock : 0.000000 [Fri Apr 12 11:14:04 2019][2937766.716077] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:04 2019][2937766.721476] .min_vruntime : 127702825.667206 [Fri Apr 12 11:14:04 2019][2937766.727569] .max_vruntime : 0.000001 [Fri Apr 12 11:14:04 2019][2937766.732967] .spread : 0.000000 [Fri Apr 12 11:14:04 2019][2937766.738368] .spread0 : 10861278.554362 [Fri Apr 12 11:14:04 2019][2937766.744375] .nr_spread_over : 0 [Fri Apr 12 11:14:04 2019][2937766.749167] .nr_running : 2 [Fri Apr 12 11:14:04 2019][2937766.753961] .load : 7168 [Fri Apr 12 11:14:04 2019][2937766.759013] .runnable_load_avg : 0 [Fri Apr 12 11:14:04 2019][2937766.763806] .blocked_load_avg : 1229 [Fri Apr 12 11:14:04 2019][2937766.768858] .tg_load_avg : 0 [Fri Apr 12 11:14:04 2019][2937766.773651] .tg_load_contrib : 0 [Fri Apr 12 11:14:04 2019][2937766.778443] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:04 2019][2937766.783236] .tg->runnable_avg : 0 [Fri Apr 12 11:14:04 2019][2937766.788028] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:04 2019][2937766.792820] .throttled : 0 [Fri Apr 12 11:14:04 2019][2937766.797614] .throttle_count : 0 [Fri Apr 12 11:14:04 2019][2937766.802408] .avg->runnable_avg_sum : 19695 [Fri Apr 12 11:14:04 2019][2937766.807545] .avg->runnable_avg_period : 46342 [Fri Apr 12 11:14:04 2019][2937766.812693] [Fri Apr 12 11:14:04 2019][2937766.814367] rt_rq[5]:/ [Fri Apr 12 11:14:04 2019][2937766.816907] .rt_nr_running : 0 [Fri Apr 12 11:14:04 2019][2937766.821699] .rt_throttled : 0 [Fri Apr 12 11:14:04 2019][2937766.826492] .rt_time : 0.010460 [Fri Apr 12 11:14:04 2019][2937766.831890] .rt_runtime : 950.000000 [Fri Apr 12 11:14:04 2019][2937766.837463] [Fri Apr 12 11:14:04 2019][2937766.839136] runnable tasks: [Fri Apr 12 11:14:04 2019][2937766.842108] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:04 2019][2937766.853056] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:05 2019][2937766.863921] watchdog/5 36 -8.966706 734428 0 0.000000 52647.840668 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.875258] migration/5 37 0.000000 47628 0 0.000000 610257.676272 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.886595] ksoftirqd/5 38 127702829.847715 44160446 120 0.000000 594919.573767 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.897933] kworker/5:0H 40 6442.228447 13 100 0.000000 0.071886 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.909318] multipathd 14514 0.000000 587531 0 0.000000 36473.746435 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.920658] multipathd 16307 0.000000 172 0 0.000000 16.268981 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.932001] kdmwork-253:3 17329 127702831.849695 259468716 120 0.000000 4603782.398427 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.943349] kworker/5:1H 18978 127702834.491585 39005490 100 0.000000 234374.057872 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.954699] kworker/5:1 13443 127702764.487064 1105276 120 0.000000 98434.937054 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.966050] ptlrpcd_01_02 117148 127701873.551159 652394 120 0.000000 16565.600018 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.977468] ptlrpcd_01_04 117150 127701873.487788 649587 120 0.000000 16228.183103 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937766.988931] ldlm_cn01_004 117682 127702835.780569 54133587 120 0.000000 1328962.893096 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.000360] ldlm_cn01_005 117687 127702849.109545 51684206 120 0.000000 1294799.045904 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.011807] mdt_out01_002 118140 127558395.618843 45 120 0.000000 1.425594 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.023231] mdt01_016 118169 129170426.606173 23618968 120 0.000000 1714658.057395 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.034659] mdt01_026 118205 117083312.280749 3971082 120 0.000000 181355.603502 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.046088] mdt01_035 118240 129170437.161038 28686280 120 0.000000 1900939.109162 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.057509] mdt01_046 118272 127702840.058740 27324371 120 0.000000 1824272.449567 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.068928] mdt01_047 118273 127583777.241022 24966916 120 0.000000 1668365.897080 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.080356] mdt01_062 118317 127702851.559313 34756406 120 0.000000 2201793.637710 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.091775] mdt01_065 118324 127624280.081486 22546903 120 0.000000 1598254.065100 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.103201] mdt01_076 118339 124024251.773970 18529072 120 0.000000 1136691.559652 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.114625] mdt01_089 118356 127579800.781244 23128994 120 0.000000 1647574.260651 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.126049] mdt01_106 118420 112338444.851417 1010625 120 0.000000 31101.912588 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.137467] mdt01_107 118423 127699711.224883 28629466 120 0.000000 1817250.616571 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.148892] mdt01_109 118437 127702470.618303 22449774 120 0.000000 1557193.047828 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.160313] mdt_io01_006 118445 127702838.929762 1271187 120 0.000000 99282.929499 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.171747] mdt_io01_007 118596 114019056.286077 131 120 0.000000 6.460759 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.183169] mdt_io01_009 118606 127702838.680009 1567566 120 0.000000 135332.158240 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.194591] mdt_out01_004 118614 127702407.760652 155331 120 0.000000 5704.620993 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.206017] mdt_io01_013 118641 127698332.143033 522560 120 0.000000 42124.886465 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.217438] mdt_io01_014 118648 114019055.028823 205804 120 0.000000 9748.265977 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.228866] mdt_io01_020 118722 114019054.961024 79 120 0.000000 5.383066 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.240294] mdt_io01_028 118774 126925550.792824 481524 120 0.000000 17957.396612 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.251724] mdt_io01_038 118840 127680267.924647 1610334 120 0.000000 152799.627790 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.263146] mdt_io01_041 118847 127683345.856542 2251303 120 0.000000 193320.407373 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.274571] mdt_io01_045 118873 118806692.969930 104 120 0.000000 5.278632 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.285992] mdt_io01_047 118882 126925550.791711 105741 120 0.000000 4899.458460 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.297425] mdt_io01_054 118950 114019054.772981 408128 120 0.000000 14964.060073 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.308851] mdt_io01_064 118997 127702876.279417 241745 120 0.000000 20418.422844 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.320270] mdt_io01_068 119009 127683345.856582 2346506 120 0.000000 181829.107179 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.331696] mdt_rdpg01_008 119036 127702369.861203 28662675 120 0.000000 1404311.287420 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.343127] mdt_rdpg01_032 119113 127702369.850933 27780394 120 0.000000 1314707.709096 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.354574] ldlm_cn01_013 123222 127702873.166198 50580516 120 0.000000 1279536.443025 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.365998] ldlm_cn01_018 123806 127702838.926255 45196872 120 0.000000 1179832.811652 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.377420] ldlm_cn01_024 126534 127702838.680660 45171376 120 0.000000 1171983.430527 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.388841] ldlm_cn01_025 126535 127702870.599440 46940057 120 0.000000 1200191.460278 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.400300] kworker/5:0 29823 121007127.503241 2 120 0.000000 0.009709 0.000000 1 / [Fri Apr 12 11:14:05 2019][2937767.411698] [Fri Apr 12 11:14:05 2019][2937767.413371] cpu#6, 1996.203 MHz [Fri Apr 12 11:14:05 2019][2937767.416692] .nr_running : 1 [Fri Apr 12 11:14:05 2019][2937767.421484] .load : 1024 [Fri Apr 12 11:14:05 2019][2937767.426538] .nr_switches : 16621338931 [Fri Apr 12 11:14:05 2019][2937767.432197] .nr_load_updates : 1979060000 [Fri Apr 12 11:14:05 2019][2937767.437777] .nr_uninterruptible : -17692 [Fri Apr 12 11:14:05 2019][2937767.443004] .next_balance : 7232.368190 [Fri Apr 12 11:14:05 2019][2937767.448663] .curr->pid : 0 [Fri Apr 12 11:14:05 2019][2937767.453457] .clock : 2937767453.407996 [Fri Apr 12 11:14:05 2019][2937767.459637] .cpu_load[0] : 0 [Fri Apr 12 11:14:05 2019][2937767.464430] .cpu_load[1] : 0 [Fri Apr 12 11:14:05 2019][2937767.469229] .cpu_load[2] : 0 [Fri Apr 12 11:14:05 2019][2937767.474023] .cpu_load[3] : 104 [Fri Apr 12 11:14:05 2019][2937767.478989] .cpu_load[4] : 77 [Fri Apr 12 11:14:05 2019][2937767.483877] .avg_idle : 198284 [Fri Apr 12 11:14:05 2019][2937767.489103] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:05 2019][2937767.494418] [Fri Apr 12 11:14:05 2019][2937767.496095] cfs_rq[6]:/ [Fri Apr 12 11:14:05 2019][2937767.498720] .exec_clock : 0.000000 [Fri Apr 12 11:14:05 2019][2937767.504119] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:05 2019][2937767.509519] .min_vruntime : 94991339.700434 [Fri Apr 12 11:14:05 2019][2937767.515525] .max_vruntime : 0.000001 [Fri Apr 12 11:14:05 2019][2937767.520925] .spread : 0.000000 [Fri Apr 12 11:14:05 2019][2937767.526323] .spread0 : -21850296.230319 [Fri Apr 12 11:14:05 2019][2937767.532416] .nr_spread_over : 0 [Fri Apr 12 11:14:05 2019][2937767.537208] .nr_running : 1 [Fri Apr 12 11:14:05 2019][2937767.542002] .load : 2048 [Fri Apr 12 11:14:05 2019][2937767.547055] .runnable_load_avg : 216 [Fri Apr 12 11:14:05 2019][2937767.552020] .blocked_load_avg : 418 [Fri Apr 12 11:14:05 2019][2937767.556988] .tg_load_avg : 0 [Fri Apr 12 11:14:05 2019][2937767.561781] .tg_load_contrib : 0 [Fri Apr 12 11:14:05 2019][2937767.566574] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:05 2019][2937767.571366] .tg->runnable_avg : 0 [Fri Apr 12 11:14:05 2019][2937767.576159] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:05 2019][2937767.580951] .throttled : 0 [Fri Apr 12 11:14:05 2019][2937767.585745] .throttle_count : 0 [Fri Apr 12 11:14:05 2019][2937767.590537] .avg->runnable_avg_sum : 13938 [Fri Apr 12 11:14:05 2019][2937767.595676] .avg->runnable_avg_period : 46376 [Fri Apr 12 11:14:05 2019][2937767.600824] [Fri Apr 12 11:14:05 2019][2937767.602496] rt_rq[6]:/ [Fri Apr 12 11:14:05 2019][2937767.605037] .rt_nr_running : 0 [Fri Apr 12 11:14:05 2019][2937767.609830] .rt_throttled : 0 [Fri Apr 12 11:14:05 2019][2937767.614623] .rt_time : 0.000000 [Fri Apr 12 11:14:05 2019][2937767.620022] .rt_runtime : 950.000000 [Fri Apr 12 11:14:05 2019][2937767.625595] [Fri Apr 12 11:14:05 2019][2937767.627265] runnable tasks: [Fri Apr 12 11:14:05 2019][2937767.630240] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:05 2019][2937767.641186] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:05 2019][2937767.652051] watchdog/6 41 -8.960795 734428 0 0.000000 31190.485493 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.663390] migration/6 42 0.000000 27685 0 0.000000 313496.899073 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.674727] ksoftirqd/6 43 94991337.399109 42559187 120 0.000000 860298.919008 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.686062] kworker/6:0H 45 4272.824385 9 100 0.000000 0.030949 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.697450] kworker/6:1H 16843 94991337.360784 24289971 100 0.000000 143083.583183 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.708799] writer#4 19295 94991261.709872 15199252 120 0.000000 236192.883927 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.720141] kworker/6:2 91769 94991302.416629 1838889 120 0.000000 48707.791002 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.731486] ptlrpc_hr02_001 117087 94991288.388528 2388695 120 0.000000 76024.213195 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.742909] ptlrpc_hr02_003 117089 94991302.417301 2388775 120 0.000000 75007.555185 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.754340] ptlrpcd_02_10 117168 94991337.323684 832895 120 0.000000 22045.515267 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.765762] ptlrpcd_02_11 117169 94991337.376145 860465 120 0.000000 21604.665795 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.777198] mdt_out02_001 117497 94987050.946625 79463 120 0.000000 3738.721890 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.788627] osp-syn-13-2 117570 94991166.555224 237588 120 0.000000 6313.912714 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.800055] osp-syn-17-2 117600 94991039.244515 238781 120 0.000000 6406.575145 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.811484] osp-syn-16-2 117612 94991039.274671 237246 120 0.000000 6474.741930 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.822913] osp-pre-8-2 117655 94991159.633075 39437 120 0.000000 1001.579930 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.834336] osp-syn-35-2 117656 94991039.287617 235953 120 0.000000 7249.693116 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.845759] osp-pre-35-2 117657 94990819.504478 40125 120 0.000000 1128.359913 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.857184] ldlm_cn02_004 117685 94991004.785176 45110265 120 0.000000 1087549.764831 0.000000 2 / [Fri Apr 12 11:14:05 2019][2937767.868609] osp-pre-46-0 117727 94991112.797675 39299 120 0.000000 983.263350 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.880030] osp-pre-31-0 117737 94991327.717186 39345 120 0.000000 872.904229 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.891454] osp-pre-32-0 117761 94990815.812858 39265 120 0.000000 946.904653 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.902882] osp-pre-35-0 117809 94990924.078127 39458 120 0.000000 907.402497 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.914343] ll_mgs_0007 117905 94991400.966011 2981676 120 0.000000 101533.705600 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.925770] mdt02_012 118174 80662166.855550 1220140 120 0.000000 40945.259878 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.937195] mdt02_013 118178 91845170.980030 15699002 120 0.000000 1004034.252831 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.948619] mdt02_016 118191 94933412.263453 15839948 120 0.000000 1240415.236036 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.960042] mdt02_022 118214 94916566.124595 16322363 120 0.000000 1166764.615214 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.971466] mdt02_026 118249 94991412.612759 17387834 120 0.000000 1074941.815889 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.982895] mdt02_030 118266 81545057.682013 2430196 120 0.000000 97873.587543 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937767.994324] mdt02_040 118336 94988958.428931 27018941 120 0.000000 1686437.145927 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.005753] mdt02_045 118365 94991198.823489 15276928 120 0.000000 1101345.511378 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.017182] mdt02_075 118489 81142809.282515 4800658 120 0.000000 188832.248541 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.028604] mdt02_076 118503 94988962.088474 13452883 120 0.000000 974477.449432 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.040028] mdt02_081 118513 94991444.508899 25762194 120 0.000000 1682974.725326 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.051451] mdt02_085 118518 94984573.003806 11621876 120 0.000000 885829.490575 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.062879] mdt02_097 118565 94984573.004117 20227720 120 0.000000 1402003.784578 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.074304] mdt02_105 118583 94984573.003756 24746725 120 0.000000 1489669.163468 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.085725] mdt02_108 118587 82165691.975017 1135415 120 0.000000 58938.991977 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.097168] mdt_io02_020 118652 94991433.957379 1733216 120 0.000000 115338.317087 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.108602] mdt_io02_027 118680 94942898.057772 286611 120 0.000000 21196.799083 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.120021] mdt_io02_030 118689 94498590.309910 699982 120 0.000000 55025.415264 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.131468] mdt_io02_045 118744 81940911.279325 86930 120 0.000000 3584.136780 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.142895] mdt_io02_048 118757 94991437.587649 1751623 120 0.000000 156825.918687 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.154340] mdt_out02_002 118902 94905090.876444 132944 120 0.000000 5159.446423 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.165768] mdt_io02_093 118931 94942898.056930 1192708 120 0.000000 123846.183826 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.177217] mdt_rdpg02_029 119121 94989608.003565 27646585 120 0.000000 1287222.453289 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.188642] mdt_rdpg02_041 119167 94989608.006642 17928001 120 0.000000 942936.535118 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.200095] ldlm_cn02_014 126949 94991004.785206 37706190 120 0.000000 937202.540473 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.211517] ldlm_cn02_015 126950 94991266.827728 40936082 120 0.000000 1021717.401522 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.222960] osp-pre-41-1 2437 94991174.474169 27702 120 0.000000 719.046633 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.234298] lquota_wb_fir-M 2506 94989781.166933 2339 120 0.000000 40.745284 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.245636] ldlm_cn02_028 6494 94991264.139528 33557659 120 0.000000 856076.569383 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.256988] kworker/6:0 45707 94718272.166856 2 120 0.000000 0.008636 0.000000 2 / [Fri Apr 12 11:14:06 2019][2937768.268356] [Fri Apr 12 11:14:06 2019][2937768.270049] cpu#7, 1996.203 MHz [Fri Apr 12 11:14:06 2019][2937768.273380] .nr_running : 0 [Fri Apr 12 11:14:06 2019][2937768.278178] .load : 0 [Fri Apr 12 11:14:06 2019][2937768.282973] .nr_switches : 11602601042 [Fri Apr 12 11:14:06 2019][2937768.288650] .nr_load_updates : 1819347870 [Fri Apr 12 11:14:06 2019][2937768.294231] .nr_uninterruptible : -26607 [Fri Apr 12 11:14:06 2019][2937768.299495] .next_balance : 7232.369047 [Fri Apr 12 11:14:06 2019][2937768.305160] .curr->pid : 0 [Fri Apr 12 11:14:06 2019][2937768.309960] .clock : 2937768309.942136 [Fri Apr 12 11:14:06 2019][2937768.316150] .cpu_load[0] : 1024 [Fri Apr 12 11:14:06 2019][2937768.321210] .cpu_load[1] : 592 [Fri Apr 12 11:14:06 2019][2937768.326192] .cpu_load[2] : 212 [Fri Apr 12 11:14:06 2019][2937768.331159] .cpu_load[3] : 131 [Fri Apr 12 11:14:06 2019][2937768.336125] .cpu_load[4] : 180 [Fri Apr 12 11:14:06 2019][2937768.341099] .avg_idle : 271464 [Fri Apr 12 11:14:06 2019][2937768.346327] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:06 2019][2937768.351675] [Fri Apr 12 11:14:06 2019][2937768.353352] cfs_rq[7]:/ [Fri Apr 12 11:14:06 2019][2937768.355978] .exec_clock : 0.000000 [Fri Apr 12 11:14:06 2019][2937768.361376] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:06 2019][2937768.366777] .min_vruntime : 80567029.707863 [Fri Apr 12 11:14:06 2019][2937768.372783] .max_vruntime : 0.000001 [Fri Apr 12 11:14:06 2019][2937768.378182] .spread : 0.000000 [Fri Apr 12 11:14:06 2019][2937768.383582] .spread0 : -36274691.561134 [Fri Apr 12 11:14:06 2019][2937768.389674] .nr_spread_over : 0 [Fri Apr 12 11:14:06 2019][2937768.394465] .nr_running : 0 [Fri Apr 12 11:14:06 2019][2937768.399261] .load : 0 [Fri Apr 12 11:14:06 2019][2937768.404053] .runnable_load_avg : 0 [Fri Apr 12 11:14:06 2019][2937768.408844] .blocked_load_avg : 62 [Fri Apr 12 11:14:06 2019][2937768.413723] .tg_load_avg : 0 [Fri Apr 12 11:14:06 2019][2937768.418518] .tg_load_contrib : 0 [Fri Apr 12 11:14:06 2019][2937768.423310] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:06 2019][2937768.428102] .tg->runnable_avg : 0 [Fri Apr 12 11:14:06 2019][2937768.432894] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:06 2019][2937768.437688] .throttled : 0 [Fri Apr 12 11:14:06 2019][2937768.442481] .throttle_count : 0 [Fri Apr 12 11:14:06 2019][2937768.447275] .avg->runnable_avg_sum : 3256 [Fri Apr 12 11:14:06 2019][2937768.452325] .avg->runnable_avg_period : 46246 [Fri Apr 12 11:14:06 2019][2937768.457474] [Fri Apr 12 11:14:06 2019][2937768.459148] rt_rq[7]:/ [Fri Apr 12 11:14:06 2019][2937768.461686] .rt_nr_running : 0 [Fri Apr 12 11:14:06 2019][2937768.466480] .rt_throttled : 0 [Fri Apr 12 11:14:06 2019][2937768.471273] .rt_time : 0.024276 [Fri Apr 12 11:14:06 2019][2937768.476671] .rt_runtime : 950.000000 [Fri Apr 12 11:14:06 2019][2937768.482253] [Fri Apr 12 11:14:06 2019][2937768.483925] runnable tasks: [Fri Apr 12 11:14:06 2019][2937768.486898] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:06 2019][2937768.497847] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:06 2019][2937768.508710] watchdog/7 46 -8.960245 734428 0 0.000000 22526.882342 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.520056] migration/7 47 0.000000 7812 0 0.000000 68880.670121 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.531401] ksoftirqd/7 48 80566906.502015 16067865 120 0.000000 174137.285507 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.542738] kworker/7:0H 50 4289.243881 9 100 0.000000 0.195782 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.554107] poll_mpt3sas0_h 5276 846.151895 2 100 0.000000 0.010039 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.565461] multipathd 16306 0.000000 5873498 0 0.000000 541473.777397 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.576808] polkitd 18019 80566759.197276 738034 120 0.000000 35006.136751 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.588155] rpcbind 18022 80565876.047648 97828 120 0.000000 3087.202145 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.599504] kworker/7:1H 20551 80567005.249688 14171195 100 0.000000 71971.882336 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.610848] kworker/7:1 39505 80567005.054336 2025290 120 0.000000 32668.530894 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.622194] python 39148 80566979.775144 2084441 120 0.000000 4040267.960844 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.633538] cfs_rh_00 117052 80566848.276593 9860 120 0.000000 6397.168100 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.644971] monitor_thread 117131 80566991.976868 185590 120 0.000000 26325.513386 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.656405] ptlrpcd_rcv 117133 80562532.037500 872 120 0.000000 32.997018 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.667833] ptlrpcd_03_09 117179 80567028.235351 2363875 120 0.000000 52467.922222 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.679261] ldlm_cb03_000 117264 80496669.754445 1057 120 0.000000 37.380381 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.690684] kmmpd-dm-0 117445 80566906.503578 74245 120 0.000000 1137.754146 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.702120] osp-pre-46-2 117575 80566862.073902 39815 120 0.000000 891.625079 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.713556] osp-pre-2-2 117591 80566978.172887 40149 120 0.000000 963.978191 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.724980] osp-pre-18-2 117597 80566787.115514 40292 120 0.000000 854.793206 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.736408] osp-pre-24-2 117633 80566848.749441 41757 120 0.000000 1052.371102 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.747833] osp-pre-38-2 117637 80567032.215914 39927 120 0.000000 977.871377 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.759255] osp-syn-1-2 117642 80566810.818707 235504 120 0.000000 6483.476489 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.770689] osp-syn-36-2 117652 80566810.809237 235419 120 0.000000 6602.188291 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.782121] osp-syn-2-0 117742 80551218.102909 75524 120 0.000000 2187.577994 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.793564] osp-pre-23-0 117771 80566889.034221 40046 120 0.000000 1005.713067 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.804991] osp-syn-7-0 117776 80550950.883046 75003 120 0.000000 2289.388180 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.816417] osp-pre-27-0 117801 80566875.615441 41044 120 0.000000 1111.798050 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.827837] osp-pre-28-0 117811 80566795.820036 39284 120 0.000000 920.858521 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.839258] osp-pre-34-0 117815 80566878.285849 38684 120 0.000000 753.465072 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.850702] mdt03_019 118255 80515994.210708 2943649 120 0.000000 169836.695170 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.862135] mdt03_022 118291 80567046.725327 7348463 120 0.000000 494037.526947 0.000000 3 / [Fri Apr 12 11:14:06 2019][2937768.873561] mdt03_025 118300 80508072.375007 7356311 120 0.000000 581508.314582 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.885004] mdt03_038 118415 80566979.616993 8379801 120 0.000000 654947.445818 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.896428] mdt03_043 118452 80566979.786376 18662079 120 0.000000 1093204.943195 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.907850] mdt03_048 118465 80521497.466040 5655125 120 0.000000 273579.888869 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.919279] mdt03_058 118562 80563044.459472 6747052 120 0.000000 298571.201377 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.930703] mdt_rdpg03_025 118566 80547920.372724 14957153 120 0.000000 632621.778592 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.942133] mdt03_073 118644 58774435.627918 242488 120 0.000000 9117.021649 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.953567] mdt_io03_014 118654 63513140.064294 71 120 0.000000 2.785368 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.964992] mdt03_081 118682 80566260.037885 16425119 120 0.000000 883815.856349 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.976412] mdt03_086 118704 80524393.036781 18794499 120 0.000000 1181431.966823 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.987835] mdt03_087 118709 80561872.357041 20465274 120 0.000000 1080431.258284 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937768.999289] mdt03_091 118721 80566281.793574 16128579 120 0.000000 831178.201908 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.010715] mdt03_092 118724 80533980.905722 18588010 120 0.000000 1057134.933099 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.022138] mdt_io03_030 118725 63513194.253601 523484 120 0.000000 20014.624926 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.033564] mdt03_098 118753 80567048.593651 729844 120 0.000000 48526.485332 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.044986] mdt_io03_044 118786 63513139.970783 37 120 0.000000 1.636948 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.056408] mdt03_109 118796 80566281.793513 6705368 120 0.000000 328595.759403 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.067843] mdt_io03_049 118808 63513140.038552 111355 120 0.000000 4080.783261 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.079289] mdt_io03_054 118823 63513140.015459 42 120 0.000000 2.552508 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.090713] mdt_io03_055 118829 63513140.189249 174004 120 0.000000 6654.086448 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.102144] mdt_io03_071 118881 63513140.022542 36 120 0.000000 1.782343 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.113613] mdt_rdpg03_035 119173 80547920.370890 14691060 120 0.000000 538419.959966 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.125052] mdt_rdpg03_053 119261 80564514.411921 17713187 120 0.000000 759025.657690 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.136469] mdt_rdpg03_054 119262 80560075.412114 16298829 120 0.000000 633935.656595 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.147895] ldlm_cn03_006 119290 80566344.499276 48366697 120 0.000000 932961.854233 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.159317] ldlm_cn03_008 119787 80567039.713511 45861476 120 0.000000 884192.800240 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.170748] ldlm_bl_05 125543 80566991.689775 805745 120 0.000000 14985.071623 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.182177] ldlm_bl_10 128943 80566979.599991 699608 120 0.000000 46516.214869 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.193594] ldlm_bl_11 128944 80566993.439949 702598 120 0.000000 44544.054083 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.205026] osp-pre-2-3 2255 80566861.836702 27879 120 0.000000 698.616059 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.216363] osp-syn-5-3 2268 80566677.722677 9541 120 0.000000 292.153625 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.227718] osp-pre-32-3 2273 80566983.745652 28013 120 0.000000 506.121401 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.239063] osp-pre-4-3 2281 80566979.594633 28012 120 0.000000 533.520011 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.250418] osp-syn-13-1 2392 80565192.733152 20461 120 0.000000 494.747066 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.261779] osp-syn-17-1 2422 80565194.858497 20221 120 0.000000 489.755823 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.273129] osp-pre-17-1 2423 80567010.185193 27516 120 0.000000 580.331509 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.284485] osp-pre-32-1 2431 80566869.402602 27879 120 0.000000 623.283867 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.295820] osp-syn-4-1 2438 80565105.044289 19549 120 0.000000 527.142711 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.307210] osp-syn-23-1 2440 80565389.695681 20345 120 0.000000 479.720773 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.318554] osp-syn-7-1 2446 80565108.962275 19667 120 0.000000 497.639676 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.329890] osp-pre-7-1 2447 80566893.942744 27842 120 0.000000 642.689565 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.341230] osp-pre-34-1 2485 80566898.191193 27863 120 0.000000 636.783287 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.352563] mdt_io03_086 3056 63513151.821811 6 120 0.000000 0.418074 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.363898] mdt_io03_087 3057 63513151.889420 14 120 0.000000 0.629568 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.375235] mdt_io03_091 3061 63513144.741054 8 120 0.000000 0.311532 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.386583] ldlm_cn03_022 44379 80566981.684504 1692397 120 0.000000 48053.701463 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.397915] ldlm_cn03_025 44382 83624009.741808 1245173 120 0.000000 35163.299170 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.409256] kworker/7:2 46131 80432743.467593 2 120 0.000000 0.007555 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.420597] pickup 46136 80566460.973894 25 120 0.000000 9.007328 0.000000 3[2937769.431461] LustreError: dumping log to /tmp/lustre-log.1555092847.118421 [Fri Apr 12 11:14:07 2019] / [Fri Apr 12 11:14:07 2019][2937769.438999] kworker/u256:1 46414 80567047.427219 2515 120 0.000000 44.500952 0.000000 3 / [Fri Apr 12 11:14:07 2019][2937769.450365] [Fri Apr 12 11:14:07 2019][2937769.452062] cpu#8, 1996.203 MHz [Fri Apr 12 11:14:07 2019][2937769.455402] .nr_running : 0 [Fri Apr 12 11:14:07 2019][2937769.460205] .load : 0 [Fri Apr 12 11:14:07 2019][2937769.465011] .nr_switches : 19579240820 [Fri Apr 12 11:14:07 2019][2937769.470680] .nr_load_updates : 2135226026 [Fri Apr 12 11:14:07 2019][2937769.476260] .nr_uninterruptible : 684156 [Fri Apr 12 11:14:07 2019][2937769.481485] .next_balance : 7232.370227 [Fri Apr 12 11:14:07 2019][2937769.487144] .curr->pid : 0 [Fri Apr 12 11:14:07 2019][2937769.491937] .clock : 2937769491.811615 [Fri Apr 12 11:14:07 2019][2937769.498117] .cpu_load[0] : 0 [Fri Apr 12 11:14:07 2019][2937769.502911] .cpu_load[1] : 648 [Fri Apr 12 11:14:07 2019][2937769.507876] .cpu_load[2] : 888 [Fri Apr 12 11:14:07 2019][2937769.512842] .cpu_load[3] : 837 [Fri Apr 12 11:14:07 2019][2937769.517808] .cpu_load[4] : 733 [Fri Apr 12 11:14:07 2019][2937769.522793] .avg_idle : 163858 [Fri Apr 12 11:14:07 2019][2937769.528025] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:07 2019][2937769.533338] [Fri Apr 12 11:14:07 2019][2937769.535008] cfs_rq[8]:/ [Fri Apr 12 11:14:07 2019][2937769.537635] .exec_clock : 0.000000 [Fri Apr 12 11:14:07 2019][2937769.543034] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:07 2019][2937769.548435] .min_vruntime : 116864421.348458 [Fri Apr 12 11:14:07 2019][2937769.554528] .max_vruntime : 0.000001 [Fri Apr 12 11:14:07 2019][2937769.559926] .spread : 0.000000 [Fri Apr 12 11:14:07 2019][2937769.565326] .spread0 : 22539.060256 [Fri Apr 12 11:14:07 2019][2937769.571071] .nr_spread_over : 0 [Fri Apr 12 11:14:07 2019][2937769.575863] .nr_running : 0 [Fri Apr 12 11:14:07 2019][2937769.580659] .load : 0 [Fri Apr 12 11:14:07 2019][2937769.585460] .runnable_load_avg : 0 [Fri Apr 12 11:14:07 2019][2937769.590252] .blocked_load_avg : 245 [Fri Apr 12 11:14:07 2019][2937769.595219] .tg_load_avg : 0 [Fri Apr 12 11:14:07 2019][2937769.600009] .tg_load_contrib : 0 [Fri Apr 12 11:14:07 2019][2937769.604804] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:07 2019][2937769.609594] .tg->runnable_avg : 0 [Fri Apr 12 11:14:07 2019][2937769.614390] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:07 2019][2937769.619182] .throttled : 0 [Fri Apr 12 11:14:07 2019][2937769.623976] .throttle_count : 0 [Fri Apr 12 11:14:07 2019][2937769.628768] .avg->runnable_avg_sum : 13315 [Fri Apr 12 11:14:07 2019][2937769.633907] .avg->runnable_avg_period : 46609 [Fri Apr 12 11:14:07 2019][2937769.639071] [Fri Apr 12 11:14:07 2019][2937769.640745] rt_rq[8]:/ [Fri Apr 12 11:14:07 2019][2937769.643284] .rt_nr_running : 0 [Fri Apr 12 11:14:07 2019][2937769.648094] .rt_throttled : 0 [Fri Apr 12 11:14:07 2019][2937769.652888] .rt_time : 0.000000 [Fri Apr 12 11:14:07 2019][2937769.658296] .rt_runtime : 950.000000 [Fri Apr 12 11:14:07 2019][2937769.663868] [Fri Apr 12 11:14:07 2019][2937769.665541] runnable tasks: [Fri Apr 12 11:14:07 2019][2937769.668514] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:07 2019][2937769.679460] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:07 2019][2937769.690334] watchdog/8 51 -11.974461 734428 0 0.000000 36997.591609 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.701671] migration/8 52 0.000000 27510 0 0.000000 359096.917222 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.713009] ksoftirqd/8 53 116864319.638256 25576419 120 0.000000 367969.889818 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.724354] kworker/8:0H 55 3462.592581 9 100 0.000000 0.031490 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.735738] scsi_eh_2 7254 1317.937012 22 120 0.000000 5.632576 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.747114] ccp-5-q3 13563 2639.418044 2 120 0.000000 0.010820 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.758459] ccp-5-q4 13565 2651.425832 2 120 0.000000 0.007794 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.769812] kworker/8:1H 16830 116864319.605121 22773080 100 0.000000 149704.629407 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.781160] dsm_sa_datamgrd 19157 116864378.482428 3640065 120 0.000000 97004.788389 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.792519] ptlrpcd_00_07 117141 116864155.230789 879666 120 0.000000 22442.794063 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.803949] ldlm_cn00_000 117266 116864431.251829 41542384 120 0.000000 979023.239039 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.815383] ldlm_cn00_001 117267 116864017.508080 42124001 120 0.000000 998817.029233 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.826816] mdt00_001 117464 116823647.563505 29971040 120 0.000000 1544792.373154 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.838236] mdt_rdpg00_000 117475 116864423.011955 21205245 120 0.000000 940940.532585 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.849668] mdt00_006 117560 100993988.348508 4282636 120 0.000000 158347.638866 0.000000 0 / [Fri Apr 12 11:14:07 2019][2937769.861092] osp-syn-26-2 117582 116863794.612789 236438 120 0.000000 6564.853549 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.872513] osp-pre-31-2 117585 116863944.386697 41073 120 0.000000 962.956976 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.883940] osp-syn-44-2 117606 116864228.673040 235764 120 0.000000 6648.524705 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.895363] osp-syn-45-2 117628 116863794.178674 235719 120 0.000000 6250.610980 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.906795] osp-pre-10-0 117729 116864220.308780 39254 120 0.000000 997.573856 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.918237] mdt00_009 118153 111718673.076872 16880998 120 0.000000 943942.682650 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.929682] mdt00_037 118243 116864432.291753 14601188 120 0.000000 910606.283628 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.941106] mdt00_047 118274 116859917.144945 20904861 120 0.000000 1240791.173187 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.952528] mdt00_053 118292 116859917.145395 14513720 120 0.000000 924577.733401 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.963951] mdt00_054 118294 116779825.193266 21992973 120 0.000000 1358047.288026 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.975380] mdt00_063 118363 116862132.913432 27100414 120 0.000000 1388268.848804 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.986807] mdt00_070 118399 116780102.056118 23539351 120 0.000000 1486807.395015 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937769.998238] mdt00_084 118455 99837369.544557 2903383 120 0.000000 98183.428034 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.009658] mdt00_087 118469 105694151.920254 6726277 120 0.000000 320831.984567 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.021088] mdt00_088 118471 99856943.454805 3261112 120 0.000000 110823.599848 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.032529] mdt00_096 118548 111396602.824914 18156614 120 0.000000 991128.999916 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.043967] mdt_io00_028 118674 116829193.092597 2428837 120 0.000000 178178.851760 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.055413] mdt_io00_040 118773 101341558.075625 1117 120 0.000000 28.980537 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.066835] mdt_io00_044 118792 101341558.020320 76477 120 0.000000 2451.415951 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.078290] mdt_io00_092 119011 101341324.433191 231 120 0.000000 4.879430 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.089725] mdt_rdpg00_010 119028 116864122.486297 24092600 120 0.000000 1110133.433600 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.101146] mdt_rdpg00_011 119030 116864122.486717 17677851 120 0.000000 930613.892190 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.112573] mdt_io00_101 119047 101341557.875595 783130 120 0.000000 38267.790660 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.124005] mdt_rdpg00_018 119073 116862531.675304 22709409 120 0.000000 1065381.690334 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.135435] mdt_rdpg00_021 119089 116858277.233961 20949809 120 0.000000 1019886.196471 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.146863] mdt_rdpg00_029 119137 116864414.226848 16441215 120 0.000000 777432.220286 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.158286] mdt_rdpg00_030 119143 116858277.233610 21471272 120 0.000000 1008479.752274 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.169727] mdt_io00_111 119144 101341324.885270 2898 120 0.000000 69.769331 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.181169] mdt_rdpg00_036 119161 116862531.675344 21957428 120 0.000000 948754.543787 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.192610] mdt_rdpg00_039 119181 116863048.780285 24040673 120 0.000000 1003544.484622 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.204032] mdt_rdpg00_042 119196 116863048.780255 21293160 120 0.000000 937392.486288 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.215456] mdt_rdpg00_048 119219 116864464.162340 17079972 120 0.000000 813122.590738 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.226888] mdt_rdpg00_052 119235 116864122.486246 24771500 120 0.000000 1020328.465310 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.238359] ldlm_cn00_015 126538 116864017.507669 35529431 120 0.000000 880641.636737 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.249792] ldlm_cn00_019 127576 116864311.844234 39244986 120 0.000000 945439.038929 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.261225] ldlm_cn00_022 129945 116862531.674943 34533951 120 0.000000 858718.240754 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.272668] osp-pre-28-3 2323 116864155.451659 27858 120 0.000000 600.204357 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.284020] ldlm_cn00_029 6486 116864311.844775 33134737 120 0.000000 808678.912462 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.295355] kworker/8:1 15011 116485578.726416 23712 120 0.000000 1451.922507 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.306711] kworker/8:0 34751 116864465.856649 53993 120 0.000000 3079.094603 0.000000 0 / [Fri Apr 12 11:14:08 2019][2937770.318071] [Fri Apr 12 11:14:08 2019][2937770.319744] cpu#9, 1996.203 MHz [Fri Apr 12 11:14:08 2019][2937770.323064] .nr_running : 0 [Fri Apr 12 11:14:08 2019][2937770.327861] .load : 0 [Fri Apr 12 11:14:08 2019][2937770.332707] .nr_switches : 20356786364 [Fri Apr 12 11:14:08 2019][2937770.338373] .nr_load_updates : 2188593081 [Fri Apr 12 11:14:08 2019][2937770.343955] .nr_uninterruptible : -77146 [Fri Apr 12 11:14:08 2019][2937770.349199] .next_balance : 7232.371094 [Fri Apr 12 11:14:08 2019][2937770.354875] .curr->pid : 118323 [Fri Apr 12 11:14:08 2019][2937770.360108] .clock : 2937770359.461468 [Fri Apr 12 11:14:08 2019][2937770.366296] .cpu_load[0] : 0 [Fri Apr 12 11:14:08 2019][2937770.371127] .cpu_load[1] : 308 [Fri Apr 12 11:14:08 2019][2937770.376107] .cpu_load[2] : 637 [Fri Apr 12 11:14:08 2019][2937770.381083] .cpu_load[3] : 1337 [Fri Apr 12 11:14:08 2019][2937770.386144] .cpu_load[4] : 1332 [Fri Apr 12 11:14:08 2019][2937770.391206] .avg_idle : 92610 [Fri Apr 12 11:14:08 2019][2937770.396352] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:08 2019][2937770.401691] [Fri Apr 12 11:14:08 2019][2937770.403368] cfs_rq[9]:/ [Fri Apr 12 11:14:08 2019][2937770.405999] .exec_clock : 0.000000 [Fri Apr 12 11:14:08 2019][2937770.411402] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:08 2019][2937770.416801] .min_vruntime : 131771969.864403 [Fri Apr 12 11:14:08 2019][2937770.422904] .max_vruntime : 0.000001 [Fri Apr 12 11:14:08 2019][2937770.428322] .spread : 0.000000 [Fri Apr 12 11:14:08 2019][2937770.433721] .spread0 : 14930025.765819 [Fri Apr 12 11:14:08 2019][2937770.439736] .nr_spread_over : 0 [Fri Apr 12 11:14:08 2019][2937770.444526] .nr_running : 1 [Fri Apr 12 11:14:08 2019][2937770.449321] .load : 1024 [Fri Apr 12 11:14:08 2019][2937770.454375] .runnable_load_avg : 383 [Fri Apr 12 11:14:08 2019][2937770.459340] .blocked_load_avg : 463 [Fri Apr 12 11:14:08 2019][2937770.464304] .tg_load_avg : 0 [Fri Apr 12 11:14:08 2019][2937770.469098] .tg_load_contrib : 0 [Fri Apr 12 11:14:08 2019][2937770.473918] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:08 2019][2937770.478726] .tg->runnable_avg : 0 [Fri Apr 12 11:14:08 2019][2937770.483519] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:08 2019][2937770.488322] .throttled : 0 [Fri Apr 12 11:14:08 2019][2937770.493113] .throttle_count : 0 [Fri Apr 12 11:14:08 2019][2937770.497908] .avg->runnable_avg_sum : 26077 [Fri Apr 12 11:14:08 2019][2937770.503045] .avg->runnable_avg_period : 46710 [Fri Apr 12 11:14:08 2019][2937770.508192] [Fri Apr 12 11:14:08 2019][2937770.509867] rt_rq[9]:/ [Fri Apr 12 11:14:08 2019][2937770.512405] .rt_nr_running : 0 [Fri Apr 12 11:14:08 2019][2937770.517208] .rt_throttled : 0 [Fri Apr 12 11:14:08 2019][2937770.522011] .rt_time : 0.000000 [Fri Apr 12 11:14:08 2019][2937770.527417] .rt_runtime : 950.000000 [Fri Apr 12 11:14:08 2019][2937770.533000] [Fri Apr 12 11:14:08 2019][2937770.534670] runnable tasks: [Fri Apr 12 11:14:08 2019][2937770.537643] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:08 2019][2937770.548598] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:08 2019][2937770.559468] watchdog/9 56 -11.979841 734428 0 0.000000 53145.183877 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.570802] migration/9 57 0.000000 46114 0 0.000000 580570.296126 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.582157] ksoftirqd/9 58 131771867.928120 35731144 120 0.000000 537141.690208 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.593492] kworker/9:0H 60 3258.069399 10 100 0.000000 0.036038 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.604881] ccp-6-q1 13582 2674.156735 2 120 0.000000 0.010260 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.616251] kworker/9:1H 20548 131771867.909316 30725920 100 0.000000 202719.479916 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.627604] kworker/9:2 87683 131771917.933862 1678316 120 0.000000 62583.679959 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.638963] kiblnd_sd_01_00 117119 131772022.221856 2224381409 120 0.000000 21811252.537620 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.650478] ldlm_cb01_001 117260 122043626.259997 64 120 0.000000 1.383122 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.661912] ll_evictor 117286 131697553.041325 1622 120 0.000000 4245.270495 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.673348] mdt_out01_000 117494 131663425.991994 207551 120 0.000000 8229.609045 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.684776] mdt_io01_001 117511 122349851.023579 162 120 0.000000 6.407167 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.696251] mdt01_019 118176 131673683.771734 31183384 120 0.000000 1859232.373798 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.707680] mdt01_033 118232 131769190.727063 28040290 120 0.000000 1838164.281585 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.719108] mdt01_062 118317 131771880.059681 34757739 120 0.000000 2201982.312473 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.730547] mdt01_084 118351 131690011.780768 26421936 120 0.000000 1721201.747513 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.741987] mdt_io01_003 118412 122349851.022326 8494 120 0.000000 393.263175 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.753423] mdt_io01_004 118429 122349851.021866 12103 120 0.000000 579.703030 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.764889] mdt_io01_011 118632 122349851.022858 1182145 120 0.000000 46335.481195 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.776323] mdt_io01_019 118714 122349851.139959 1409629 120 0.000000 68895.531630 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.787774] mdt_io01_026 118759 115004359.782845 85 120 0.000000 5.065394 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.799206] mdt_io01_030 118818 122349851.022136 291193 120 0.000000 10753.097655 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.810633] mdt_io01_034 118825 122349862.234238 103144 120 0.000000 4980.374007 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.822095] mdt_io01_049 118898 122349851.022487 2429494 120 0.000000 126743.486540 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.833535] mdt_io01_051 118929 131661529.022087 2395179 120 0.000000 198494.677434 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.844967] mdt_io01_056 118963 122349851.023259 679868 120 0.000000 43686.619303 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.856395] mdt_io01_062 118973 122349851.023860 306956 120 0.000000 11288.045531 0.000000 1 / [Fri Apr 12 11:14:08 2019][2937770.867820] mdt_io01_065 119003 131661529.020084 1739613 120 0.000000 109735.988560 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.879244] mdt_rdpg01_009 119038 131772080.977433 26997976 120 0.000000 1297024.303044 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.890665] mdt_io01_076 119050 122349851.023459 1643845 120 0.000000 87075.117064 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.902112] mdt_io01_077 119052 131661529.020995 1679209 120 0.000000 138175.460962 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.913536] mdt_io01_078 119055 122349851.023529 1266275 120 0.000000 69710.510311 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.924968] mdt_io01_080 119065 122349851.021656 1721927 120 0.000000 90669.832950 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.936417] mdt_rdpg01_017 119071 131766709.862968 26365381 120 0.000000 1281385.953609 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.947841] mdt_rdpg01_020 119085 131772105.125019 22443790 120 0.000000 1191538.599248 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.959281] mdt_io01_107 119207 122349851.022597 1024218 120 0.000000 63228.314992 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.970702] mdt_io01_109 119218 115004358.922132 536119 120 0.000000 22732.316463 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.982141] ldlm_cb01_003 124105 131663423.411545 1129 120 0.000000 42.634543 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937770.993621] kworker/9:1 32709 125466826.118008 2 120 0.000000 0.009298 0.000000 1 / [Fri Apr 12 11:14:09 2019][2937771.004985] [Fri Apr 12 11:14:09 2019][2937771.006664] cpu#10, 1996.203 MHz [Fri Apr 12 11:14:09 2019][2937771.010072] .nr_running : 0 [Fri Apr 12 11:14:09 2019][2937771.014893] .load : 1024 [Fri Apr 12 11:14:09 2019][2937771.019970] .nr_switches : 24854061831 [Fri Apr 12 11:14:09 2019][2937771.025647] .nr_load_updates : 2038612887 [Fri Apr 12 11:14:09 2019][2937771.031236] .nr_uninterruptible : -111251 [Fri Apr 12 11:14:09 2019][2937771.036559] .next_balance : 7232.371783 [Fri Apr 12 11:14:09 2019][2937771.042224] .curr->pid : 0 [Fri Apr 12 11:14:09 2019][2937771.047025] .clock : 2937771047.008886 [Fri Apr 12 11:14:09 2019][2937771.053206] .cpu_load[0] : 0 [Fri Apr 12 11:14:09 2019][2937771.058018] .cpu_load[1] : 64 [Fri Apr 12 11:14:09 2019][2937771.062905] .cpu_load[2] : 435 [Fri Apr 12 11:14:09 2019][2937771.067896] .cpu_load[3] : 355 [Fri Apr 12 11:14:09 2019][2937771.072888] .cpu_load[4] : 374 [Fri Apr 12 11:14:09 2019][2937771.077875] .avg_idle : 36659 [Fri Apr 12 11:14:09 2019][2937771.083028] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:09 2019][2937771.088360] [Fri Apr 12 11:14:09 2019][2937771.090046] cfs_rq[10]:/ [Fri Apr 12 11:14:09 2019][2937771.092767] .exec_clock : 0.000000 [Fri Apr 12 11:14:09 2019][2937771.098166] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:09 2019][2937771.103564] .min_vruntime : 95305214.098551 [Fri Apr 12 11:14:09 2019][2937771.109569] .max_vruntime : 0.000001 [Fri Apr 12 11:14:09 2019][2937771.114969] .spread : 0.000000 [Fri Apr 12 11:14:09 2019][2937771.120370] .spread0 : -21536804.844947 [Fri Apr 12 11:14:09 2019][2937771.126461] .nr_spread_over : 0 [Fri Apr 12 11:14:09 2019][2937771.131255] .nr_running : 0 [Fri Apr 12 11:14:09 2019][2937771.136047] .load : 0 [Fri Apr 12 11:14:09 2019][2937771.140841] .runnable_load_avg : 0 [Fri Apr 12 11:14:09 2019][2937771.145633] .blocked_load_avg : 229 [Fri Apr 12 11:14:09 2019][2937771.150599] .tg_load_avg : 0 [Fri Apr 12 11:14:09 2019][2937771.155393] .tg_load_contrib : 0 [Fri Apr 12 11:14:09 2019][2937771.160186] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:09 2019][2937771.164978] .tg->runnable_avg : 0 [Fri Apr 12 11:14:09 2019][2937771.169772] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:09 2019][2937771.174564] .throttled : 0 [Fri Apr 12 11:14:09 2019][2937771.179357] .throttle_count : 0 [Fri Apr 12 11:14:09 2019][2937771.184150] .avg->runnable_avg_sum : 11899 [Fri Apr 12 11:14:09 2019][2937771.189288] .avg->runnable_avg_period : 46688 [Fri Apr 12 11:14:09 2019][2937771.194436] [Fri Apr 12 11:14:09 2019][2937771.196108] rt_rq[10]:/ [Fri Apr 12 11:14:09 2019][2937771.198733] .rt_nr_running : 0 [Fri Apr 12 11:14:09 2019][2937771.203527] .rt_throttled : 0 [Fri Apr 12 11:14:09 2019][2937771.208321] .rt_time : 0.014898 [Fri Apr 12 11:14:09 2019][2937771.213721] .rt_runtime : 950.000000 [Fri Apr 12 11:14:09 2019][2937771.219303] [Fri Apr 12 11:14:09 2019][2937771.220973] runnable tasks: [Fri Apr 12 11:14:09 2019][2937771.223947] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:09 2019][2937771.234893] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:09 2019][2937771.245768] watchdog/10 61 -11.978129 734429 0 0.000000 34060.572844 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.257123] migration/10 62 0.000000 37745 0 0.000000 562813.337856 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.268459] ksoftirqd/10 63 95305219.218005 25129737 120 0.000000 410372.111591 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.279797] kworker/10:0H 65 4196.602712 10 100 0.000000 0.037470 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.291208] sshd 19290 95256565.613530 1246 120 0.000000 216.162759 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.302552] kworker/10:1H 20883 95304778.552376 20162709 100 0.000000 121367.236791 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.313895] kworker/10:2 118970 95305223.277947 1884549 120 0.000000 60804.266853 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.325331] kworker/u264:2 116989 84135098.619679 198 100 0.000000 0.867369 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.336773] mdt02_000 117469 95274746.922606 15894038 120 0.000000 1252660.782544 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.348205] mdt02_003 117531 95205547.711809 19058078 120 0.000000 1378953.661777 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.359627] mdt02_007 117536 95305234.299307 18244053 120 0.000000 1185208.103193 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.371129] ll_mgs_0006 117904 95264097.818590 2669072 120 0.000000 94231.103880 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.382588] mdt02_036 118304 84145013.557433 6009161 120 0.000000 240045.276277 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.394009] mdt02_039 118319 83294228.097477 315914 120 0.000000 11982.277411 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.405435] mdt02_046 118367 87319228.794796 19817403 120 0.000000 1383048.715439 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.416858] mdt02_055 118431 95229111.057107 25900628 120 0.000000 1438310.093788 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.428277] mdt02_059 118450 95304586.034373 24111311 120 0.000000 1413671.003877 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.439708] mdt02_092 118533 95304418.295923 20607292 120 0.000000 1353051.022138 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.451136] mdt02_103 118580 95301974.196233 16068514 120 0.000000 1171196.623841 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.462557] mdt_io02_011 118615 84139283.818117 143 120 0.000000 5.662258 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.473977] mdt_io02_013 118626 84139283.781676 377974 120 0.000000 14202.578716 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.485408] mdt_io02_025 118672 84139283.101181 54477 120 0.000000 2356.933388 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.496837] mdt_io02_041 118730 84139281.266480 87 120 0.000000 5.652478 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.508272] mdt_io02_046 118748 84139718.704521 87 120 0.000000 4.163161 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.519698] mdt_io02_052 118767 84139289.073550 665626 120 0.000000 29203.384370 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.531119] mdt_io02_056 118790 84139288.568895 3561 120 0.000000 85.228094 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.542564] mdt_io02_068 118830 84139289.173860 49 120 0.000000 2.154156 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.554023] mdt_io02_086 118908 84139281.343404 84 120 0.000000 4.233959 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.565468] mdt_io02_091 118923 84139289.143542 92 120 0.000000 3.314320 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.576887] mdt_io02_092 118924 84139281.494653 558655 120 0.000000 22373.717910 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.588328] mdt_rdpg02_012 119048 95300556.315413 20382859 120 0.000000 1172554.988076 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.599768] mdt_rdpg02_040 119166 95300556.226765 21778074 120 0.000000 1172325.294128 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.611194] mdt_rdpg02_047 119191 95275817.357459 24382794 120 0.000000 1139952.991036 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.622623] mdt_rdpg02_059 119249 95305205.287590 26296789 120 0.000000 1174669.662393 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.634071] ldlm_cb02_002 122226 95190547.504250 931 120 0.000000 37.325760 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.645498] mdt_seqm_0002 123405 94944852.493365 902 120 0.000000 28.763119 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.656958] kworker/u259:1 40303 95303991.972942 35568 120 0.000000 6472.943218 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.668300] kworker/10:1 45638 94862509.403826 2 120 0.000000 0.007034 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.679636] kworker/u259:2 46727 95304781.418680 66 120 0.000000 14.755374 0.000000 2 / [Fri Apr 12 11:14:09 2019][2937771.691002] [Fri Apr 12 11:14:09 2019][2937771.692698] cpu#11, 1996.203 MHz [Fri Apr 12 11:14:09 2019][2937771.696116] .nr_running : 0 [Fri Apr 12 11:14:09 2019][2937771.700918] .load : 0 [Fri Apr 12 11:14:09 2019][2937771.705719] .nr_switches : 9870820877 [Fri Apr 12 11:14:09 2019][2937771.711315] .nr_load_updates : 1805699353 [Fri Apr 12 11:14:09 2019][2937771.716897] .nr_uninterruptible : -32647 [Fri Apr 12 11:14:09 2019][2937771.722142] .next_balance : 7232.372468 [Fri Apr 12 11:14:09 2019][2937771.727827] .curr->pid : 0 [Fri Apr 12 11:14:09 2019][2937771.732639] .clock : 2937771732.618226 [Fri Apr 12 11:14:09 2019][2937771.738868] .cpu_load[0] : 0 [Fri Apr 12 11:14:09 2019][2937771.743687] .cpu_load[1] : 128 [Fri Apr 12 11:14:09 2019][2937771.748662] .cpu_load[2] : 35 [Fri Apr 12 11:14:09 2019][2937771.753547] .cpu_load[3] : 168 [Fri Apr 12 11:14:09 2019][2937771.758514] .cpu_load[4] : 167 [Fri Apr 12 11:14:09 2019][2937771.763482] .avg_idle : 62910 [Fri Apr 12 11:14:09 2019][2937771.768660] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:09 2019][2937771.773976] [Fri Apr 12 11:14:09 2019][2937771.775646] cfs_rq[11]:/ [Fri Apr 12 11:14:09 2019][2937771.778357] .exec_clock : 0.000000 [Fri Apr 12 11:14:09 2019][2937771.783775] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:09 2019][2937771.789174] .min_vruntime : 71834670.902046 [Fri Apr 12 11:14:09 2019][2937771.795179] .max_vruntime : 0.000001 [Fri Apr 12 11:14:09 2019][2937771.800580] .spread : 0.000000 [Fri Apr 12 11:14:09 2019][2937771.805981] .spread0 : -45007411.366475 [Fri Apr 12 11:14:09 2019][2937771.812080] .nr_spread_over : 0 [Fri Apr 12 11:14:09 2019][2937771.816911] .nr_running : 0 [Fri Apr 12 11:14:09 2019][2937771.821728] .load : 0 [Fri Apr 12 11:14:09 2019][2937771.826519] .runnable_load_avg : 0 [Fri Apr 12 11:14:09 2019][2937771.831314] .blocked_load_avg : 210 [Fri Apr 12 11:14:09 2019][2937771.836279] .tg_load_avg : 0 [Fri Apr 12 11:14:09 2019][2937771.841081] .tg_load_contrib : 0 [Fri Apr 12 11:14:09 2019][2937771.845873] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:09 2019][2937771.850666] .tg->runnable_avg : 0 [Fri Apr 12 11:14:09 2019][2937771.855457] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:09 2019][2937771.860250] .throttled : 0 [Fri Apr 12 11:14:09 2019][2937771.865045] .throttle_count : 0 [Fri Apr 12 11:14:09 2019][2937771.869836] .avg->runnable_avg_sum : 4716 [Fri Apr 12 11:14:09 2019][2937771.874890] .avg->runnable_avg_period : 46570 [Fri Apr 12 11:14:10 2019][2937771.880038] [Fri Apr 12 11:14:10 2019][2937771.881711] rt_rq[11]:/ [Fri Apr 12 11:14:10 2019][2937771.884334] .rt_nr_running : 0 [Fri Apr 12 11:14:10 2019][2937771.889128] .rt_throttled : 0 [Fri Apr 12 11:14:10 2019][2937771.893920] .rt_time : 0.000000 [Fri Apr 12 11:14:10 2019][2937771.899321] .rt_runtime : 950.000000 [Fri Apr 12 11:14:10 2019][2937771.904893] [Fri Apr 12 11:14:10 2019][2937771.906565] runnable tasks: [Fri Apr 12 11:14:10 2019][2937771.909540] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:10 2019][2937771.920486] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:10 2019][2937771.931362] watchdog/11 66 -11.978089 734429 0 0.000000 24793.408205 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937771.942698] migration/11 67 0.000000 8798 0 0.000000 102062.633469 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937771.954034] ksoftirqd/11 68 71834668.800192 18940483 120 0.000000 197075.991795 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937771.965370] kworker/11:0H 70 3434.109916 9 100 0.000000 0.042532 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937771.976733] khugepaged 282 71834332.402243 510925 139 0.000000 1887211.651003 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937771.988074] scsi_tmf_1 5590 987.993772 2 100 0.000000 0.012735 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937771.999416] ccp-8-q0 13783 2767.002747 2 120 0.000000 0.011112 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.010749] ccp-8-q1 13784 2779.011548 2 120 0.000000 0.008807 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.022087] ccp-8-q2 13786 2791.020389 2 120 0.000000 0.008847 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.033424] ccp-8-q3 13787 2803.028539 2 120 0.000000 0.008156 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.044760] ccp-8-q4 13788 2815.037259 2 120 0.000000 0.008726 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.056101] kworker/11:1H 16829 71834673.163507 17729148 100 0.000000 89926.471629 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.067443] kdmwork-253:0 17105 71834673.557047 265934072 120 0.000000 4118195.254080 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.078779] kdmwork-253:4 17543 75868356.311387 32028357 120 0.000000 615776.616364 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.090132] kworker/11:0 46320 71834620.127067 1609870 120 0.000000 31351.543818 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.101482] kiblnd_sd_03_02 117129 71834687.650932 1411099187 120 0.000000 11211505.852183 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.112992] lnet_discovery 117132 71834666.708298 189577 120 0.000000 1513.819069 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.124432] mdt_out03_000 117498 71834662.325045 182022 120 0.000000 5979.855536 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.135859] mdt_io03_000 117516 56116106.035748 67 120 0.000000 1.768777 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.147279] mdt_io03_001 117517 71833922.143190 2544991 120 0.000000 185606.886136 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.158707] osp-syn-39-2 117572 71834538.467965 236031 120 0.000000 6405.969021 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.170127] osp-pre-42-2 117587 71834529.525110 42443 120 0.000000 1157.632225 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.181548] osp-pre-19-2 117595 71834426.545003 40503 120 0.000000 943.298318 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.192975] osp-syn-33-2 117620 71834526.355379 237079 120 0.000000 6454.107761 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.204400] osp-pre-22-2 117645 71834398.650941 40343 120 0.000000 864.930907 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.215826] osp-syn-3-2 117646 71834527.239789 236566 120 0.000000 6267.725476 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.227251] osp-pre-37-2 117651 71834469.335419 42529 120 0.000000 938.133108 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.238676] osp_up3-0 117718 71521303.600764 524 120 0.000000 28.182135 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.250095] osp-syn-12-0 117720 71518192.996375 75744 120 0.000000 2242.383124 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.261523] osp-syn-43-0 117744 71822565.693925 74972 120 0.000000 2312.908301 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.272951] osp-pre-17-0 117753 71834374.984263 39243 120 0.000000 794.889716 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.284373] osp-pre-0-0 117755 71834508.387534 38835 120 0.000000 792.128404 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.295797] osp-syn-21-0 117762 71822329.762998 75694 120 0.000000 2283.259332 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.307218] osp-pre-41-0 117767 71834603.203913 40151 120 0.000000 867.843148 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.318643] osp-syn-45-0 117780 71822249.912649 73570 120 0.000000 2204.615491 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.330063] osp-syn-24-0 117784 71822207.155462 74891 120 0.000000 2277.749547 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.341506] osp-pre-24-0 117785 71834365.727971 39296 120 0.000000 914.150093 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.352928] osp-pre-47-0 117787 71834571.337978 40274 120 0.000000 913.496104 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.364352] osp-syn-15-0 117790 71518192.987248 75870 120 0.000000 2350.758408 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.375775] osp-pre-37-0 117803 71834602.189906 40640 120 0.000000 832.918249 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.387204] osp-pre-36-0 117805 71834654.249142 39246 120 0.000000 857.321613 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.398659] mdt03_009 118179 70575649.871228 5541357 120 0.000000 238552.928649 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.410096] mdt03_021 118283 70575649.871238 13845161 120 0.000000 888415.179037 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.421518] mdt03_023 118295 70575649.872019 12534946 120 0.000000 871018.479077 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.432946] mdt03_030 118358 71523728.009600 3264060 120 0.000000 200702.471178 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.444395] mdt_io03_003 118418 56115965.391479 2110 120 0.000000 47.141701 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.455873] mdt_rdpg03_014 118512 71820737.227629 13745330 120 0.000000 602691.750385 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.467305] mdt_rdpg03_019 118531 71831390.103200 18835221 120 0.000000 698099.297449 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.478749] mdt_rdpg03_028 118590 71834713.660273 15014358 120 0.000000 539501.713397 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.490194] mdt_io03_016 118665 71208575.878597 933110 120 0.000000 84586.323179 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.501619] mdt_io03_021 118687 71208575.877805 1188947 120 0.000000 50846.645003 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.513039] mdt03_084 118694 71523497.986785 8286820 120 0.000000 668783.940136 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.524476] mdt_io03_031 118727 56116109.110734 480772 120 0.000000 18262.460093 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.535928] mdt_io03_033 118736 56115964.852057 42 120 0.000000 1.795064 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.547364] mdt_io03_037 118761 56115965.942808 64 120 0.000000 4.656139 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.558785] mdt_io03_042 118778 71827910.001455 1316906 120 0.000000 83060.667419 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.570231] mdt_io03_056 118833 71831152.267552 1563844 120 0.000000 105573.930267 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.581660] mdt_io03_064 118860 71827758.600895 2031684 120 0.000000 144664.228668 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.593088] mdt_io03_066 118865 56115964.388939 41 120 0.000000 1.827844 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.604604] mdt_io03_067 118867 56116108.405674 91 120 0.000000 5.670777 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.616067] mdt_io03_077 118901 56115965.102211 43 120 0.000000 2.060468 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.627512] mdt_io03_078 118904 71208575.878957 1501734 120 0.000000 138871.373603 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.638939] mdt_io03_083 118925 56115963.967807 19 120 0.000000 0.882974 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.650408] mdt_rdpg03_048 119255 71834693.253587 13369012 120 0.000000 570129.501815 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.661830] mdt_rdpg03_050 119258 71831390.103681 16242581 120 0.000000 655706.316057 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.673257] mdt_fld_0002 119277 70911806.676738 722 120 0.000000 17.018113 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.684693] ldlm_cn03_016 129487 71833606.740872 41589744 120 0.000000 806435.272340 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.696121] jbd2/dm-2-8 2218 71834671.412048 2436243 120 0.000000 224067.608157 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.707455] osp-syn-12-3 2232 71833010.966203 9704 120 0.000000 310.939399 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.718791] osp-syn-26-3 2246 71833988.621482 9437 120 0.000000 284.616523 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.730134] osp-syn-31-3 2248 71833684.105610 9561 120 0.000000 318.471702 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.741489] osp-pre-2-3 2255 71834571.248889 27880 120 0.000000 698.629134 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.752982] osp-pre-19-3 2259 71834637.106163 28041 120 0.000000 538.891536 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.764343] osp-pre-18-3 2261 71834485.513830 27873 120 0.000000 503.573532 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.775681] osp-pre-5-3 2269 71834689.197457 28011 120 0.000000 560.863044 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.787016] osp-syn-32-3 2272 71834035.759968 9464 120 0.000000 273.802693 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.798361] osp-pre-41-3 2279 71834380.228227 27997 120 0.000000 588.534523 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.809697] osp-syn-33-3 2284 71834106.541850 9538 120 0.000000 296.231533 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.821031] osp-syn-20-3 2286 71834007.869760 9489 120 0.000000 294.865469 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.832368] osp-pre-7-3 2289 71834606.050022 28027 120 0.000000 678.417495 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.843705] osp-pre-47-3 2299 71834684.321838 28034 120 0.000000 596.340331 0.000000 3 / [Fri Apr 12 11:14:10 2019][2937772.855042] osp-pre-1-3 2307 71834699.186881 28004 120 0.000000 538.903049 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.866387] osp-syn-37-3 2314 71834292.213303 9545 120 0.000000 274.834452 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.877768] osp-syn-34-3 2326 71833846.389667 9378 120 0.000000 276.396694 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.889157] osp-pre-0-1 2385 71834487.900665 27366 120 0.000000 568.206498 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.900508] osp-syn-39-1 2394 71831826.872006 20011 120 0.000000 507.155007 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.911862] osp-syn-26-1 2404 71832084.052960 19803 120 0.000000 496.544536 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.923209] osp-syn-9-1 2410 71831726.168480 19594 120 0.000000 531.613657 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.934561] osp-pre-2-1 2413 71834458.849107 27841 120 0.000000 581.624879 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.945905] osp-pre-43-1 2415 71834483.216896 27756 120 0.000000 607.664952 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.957240] osp-syn-18-1 2418 71832079.412300 20251 120 0.000000 497.490772 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.968588] osp-pre-25-1 2421 71834392.544514 27872 120 0.000000 728.009667 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.979932] osp-syn-0-1 2424 71831772.381207 19884 120 0.000000 477.953088 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937772.991275] osp-syn-44-1 2428 71832323.464135 19943 120 0.000000 491.398244 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.002639] osp-pre-21-1 2433 71834529.516763 27524 120 0.000000 515.673873 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.013992] osp-syn-16-1 2434 71832089.326110 20061 120 0.000000 493.427313 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.025330] osp-syn-6-1 2452 71832124.302892 20224 120 0.000000 511.493082 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.036681] osp-syn-47-1 2456 71832128.610413 20253 120 0.000000 506.986038 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.048018] osp-pre-1-1 2465 71834496.309368 28050 120 0.000000 671.693541 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.059364] osp-pre-37-1 2473 71834640.835155 27604 120 0.000000 638.753172 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.070699] osp-syn-36-1 2474 71831814.552942 19760 120 0.000000 524.141696 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.082036] osp-syn-35-1 2478 71832024.406854 20088 120 0.000000 505.568815 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.093371] osp-syn-28-1 2480 71832390.451641 20015 120 0.000000 463.476881 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.104717] kworker/11:1 27914 70123716.078635 3 120 0.000000 0.009087 0.000000 3 / [Fri Apr 12 11:14:11 2019][2937773.116171] [Fri Apr 12 11:14:11 2019][2937773.117852] cpu#12, 1996.203 MHz [Fri Apr 12 11:14:11 2019][2937773.121277] .nr_running : 0 [Fri Apr 12 11:14:11 2019][2937773.126078] .load : 0 [Fri Apr 12 11:14:11 2019][2937773.130888] .nr_switches : 15242382616 [Fri Apr 12 11:14:11 2019][2937773.136557] .nr_load_updates : 2120254167 [Fri Apr 12 11:14:11 2019][2937773.142139] .nr_uninterruptible : 847730 [Fri Apr 12 11:14:11 2019][2937773.147409] .next_balance : 7232.373894 [Fri Apr 12 11:14:11 2019][2937773.153102] .curr->pid : 0 [Fri Apr 12 11:14:11 2019][2937773.157902] .clock : 2937773157.890364 [Fri Apr 12 11:14:11 2019][2937773.164108] .cpu_load[0] : 1024 [Fri Apr 12 11:14:11 2019][2937773.169178] .cpu_load[1] : 298 [Fri Apr 12 11:14:11 2019][2937773.174174] .cpu_load[2] : 701 [Fri Apr 12 11:14:11 2019][2937773.179147] .cpu_load[3] : 534 [Fri Apr 12 11:14:11 2019][2937773.184138] .cpu_load[4] : 740 [Fri Apr 12 11:14:11 2019][2937773.189112] .avg_idle : 53878 [Fri Apr 12 11:14:11 2019][2937773.194269] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:11 2019][2937773.199589] [Fri Apr 12 11:14:11 2019][2937773.201266] cfs_rq[12]:/ [Fri Apr 12 11:14:11 2019][2937773.203979] .exec_clock : 0.000000 [Fri Apr 12 11:14:11 2019][2937773.209381] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:11 2019][2937773.214788] .min_vruntime : 108726286.962996 [Fri Apr 12 11:14:11 2019][2937773.220878] .max_vruntime : 0.000001 [Fri Apr 12 11:14:11 2019][2937773.226279] .spread : 0.000000 [Fri Apr 12 11:14:11 2019][2937773.231697] .spread0 : -8115940.243372 [Fri Apr 12 11:14:11 2019][2937773.237737] .nr_spread_over : 0 [Fri Apr 12 11:14:11 2019][2937773.242538] .nr_running : 3 [Fri Apr 12 11:14:11 2019][2937773.247332] .load : 0 [Fri Apr 12 11:14:11 2019][2937773.252141] .runnable_load_avg : 0 [Fri Apr 12 11:14:11 2019][2937773.256935] .blocked_load_avg : 735 [Fri Apr 12 11:14:11 2019][2937773.261909] .tg_load_avg : 0 [Fri Apr 12 11:14:11 2019][2937773.266701] .tg_load_contrib : 0 [Fri Apr 12 11:14:11 2019][2937773.271501] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:11 2019][2937773.276296] .tg->runnable_avg : 0 [Fri Apr 12 11:14:11 2019][2937773.281089] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:11 2019][2937773.285890] .throttled : 0 [Fri Apr 12 11:14:11 2019][2937773.290683] .throttle_count : 0 [Fri Apr 12 11:14:11 2019][2937773.295476] .avg->runnable_avg_sum : 11877 [Fri Apr 12 11:14:11 2019][2937773.300625] .avg->runnable_avg_period : 46764 [Fri Apr 12 11:14:11 2019][2937773.305772] [Fri Apr 12 11:14:11 2019][2937773.307445] rt_rq[12]:/ [Fri Apr 12 11:14:11 2019][2937773.310070] .rt_nr_running : 0 [Fri Apr 12 11:14:11 2019][2937773.314864] .rt_throttled : 0 [Fri Apr 12 11:14:11 2019][2937773.319657] .rt_time : 0.000000 [Fri Apr 12 11:14:11 2019][2937773.325056] .rt_runtime : 950.000000 [Fri Apr 12 11:14:11 2019][2937773.330628] [Fri Apr 12 11:14:11 2019][2937773.332300] runnable tasks: [Fri Apr 12 11:14:11 2019][2937773.335273] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:11 2019][2937773.346220] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:11 2019][2937773.357087] watchdog/12 71 -11.981345 734429 0 0.000000 36054.678376 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.368425] migration/12 72 0.000000 33601 0 0.000000 497113.958848 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.379769] ksoftirqd/12 73 108725981.486468 29856869 120 0.000000 412951.006642 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.391105] kworker/12:0H 75 3419.660109 9 100 0.000000 0.026141 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.402483] ccp-6-q3 13601 2472.450169 2 120 0.000000 0.007364 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.413851] kworker/12:1H 17445 108725981.475578 27723014 100 0.000000 175973.764450 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.425205] kworker/12:0 24475 106483586.858230 1980406 120 0.000000 36791.719658 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.436554] ptlrpc_hr00_003 117076 108726279.498970 2395330 120 0.000000 69915.009935 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.447982] ptlrpcd_00_00 117134 108725976.667174 848727 120 0.000000 23097.133968 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.459451] osp-syn-28-2 117658 108726259.380210 236042 120 0.000000 6906.649547 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.470875] osp-pre-28-2 117659 108726259.354532 40680 120 0.000000 1087.538982 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.482313] osp-syn-4-0 117768 108688403.550609 75454 120 0.000000 2238.590502 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.493759] mdt00_020 118186 108721623.481921 20596578 120 0.000000 1335533.147223 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.505190] mdt00_050 118282 108726283.777904 23894881 120 0.000000 1231944.834760 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.516621] mdt00_071 118407 108716043.886216 23257173 120 0.000000 1258764.107618 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.528041] mdt_io00_003 118411 108684166.076387 2911608 120 0.000000 191250.108855 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.539473] mdt00_077 118430 108701046.818494 24575643 120 0.000000 1207887.841926 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.550906] mdt00_078 118435 91331271.929981 64349 120 0.000000 2519.008728 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.562327] mdt00_082 118449 108716043.881887 26395136 120 0.000000 1461150.667977 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.573752] mdt00_086 118468 108646692.083003 22628813 120 0.000000 1383299.932370 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.585191] mdt00_091 118482 92323538.746757 705415 120 0.000000 24181.510821 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.596632] mdt_io00_025 118661 93225694.613205 158055 120 0.000000 5468.586005 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.608061] mdt_io00_034 118729 108684166.076266 2578876 120 0.000000 206503.989295 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.619491] mdt_io00_042 118784 93225692.052781 103 120 0.000000 5.115640 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.630925] mdt_io00_050 118828 93225482.374214 75 120 0.000000 3.691268 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.642362] mdt_io00_065 118890 93225481.228694 38 120 0.000000 1.985432 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.653791] mdt_io00_079 118958 108684166.076888 2388723 120 0.000000 209136.702455 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.665217] mdt_io00_089 119004 108684166.076627 3308960 120 0.000000 249872.492477 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.676654] mdt_io00_102 119053 93225682.389683 43 120 0.000000 3.058510 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.688105] mdt_rdpg00_053 119237 108722155.267801 27383323 120 0.000000 1176452.962330 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.699534] ldlm_cn00_008 119915 108722155.268082 42829899 120 0.000000 1021427.145135 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.710966] ldlm_cn00_017 127574 108726399.551936 36898688 120 0.000000 888689.445781 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.722396] osp-pre-31-3 2249 108725953.392258 28024 120 0.000000 616.598782 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.733756] kworker/u257:2 34641 108726105.238067 306312 120 0.000000 50278.365966 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.745096] kworker/12:1 37779 108726393.389175 69806 120 0.000000 4809.211413 0.000000 0 / [Fri Apr 12 11:14:11 2019][2937773.756449] [Fri Apr 12 11:14:11 2019][2937773.758126] cpu#13, 1996.203 MHz [Fri Apr 12 11:14:11 2019][2937773.761549] .nr_running : 0 [Fri Apr 12 11:14:11 2019][2937773.766368] .load : 1024 [Fri Apr 12 11:14:11 2019][2937773.771429] .nr_switches : 18244545464 [Fri Apr 12 11:14:11 2019][2937773.777098] .nr_load_updates : 2167889890 [Fri Apr 12 11:14:11 2019][2937773.782681] .nr_uninterruptible : -87618 [Fri Apr 12 11:14:11 2019][2937773.787918] .next_balance : 7232.374535 [Fri Apr 12 11:14:11 2019][2937773.793607] .curr->pid : 0 [Fri Apr 12 11:14:11 2019][2937773.798419] .clock : 2937773798.372730 [Fri Apr 12 11:14:11 2019][2937773.804607] .cpu_load[0] : 0 [Fri Apr 12 11:14:11 2019][2937773.809411] .cpu_load[1] : 22 [Fri Apr 12 11:14:11 2019][2937773.814297] .cpu_load[2] : 222 [Fri Apr 12 11:14:11 2019][2937773.819269] .cpu_load[3] : 112 [Fri Apr 12 11:14:11 2019][2937773.824236] .cpu_load[4] : 285 [Fri Apr 12 11:14:11 2019][2937773.829201] .avg_idle : 293582 [Fri Apr 12 11:14:11 2019][2937773.834431] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:11 2019][2937773.839788] [Fri Apr 12 11:14:11 2019][2937773.841464] cfs_rq[13]:/ [Fri Apr 12 11:14:11 2019][2937773.844176] .exec_clock : 0.000000 [Fri Apr 12 11:14:11 2019][2937773.849584] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:11 2019][2937773.854984] .min_vruntime : 129171377.808641 [Fri Apr 12 11:14:11 2019][2937773.861078] .max_vruntime : 0.000001 [Fri Apr 12 11:14:11 2019][2937773.866474] .spread : 0.000000 [Fri Apr 12 11:14:12 2019][2937773.871884] .spread0 : 12329109.134343 [Fri Apr 12 11:14:12 2019][2937773.877905] .nr_spread_over : 0 [Fri Apr 12 11:14:12 2019][2937773.882699] .nr_running : 0 [Fri Apr 12 11:14:12 2019][2937773.887491] .load : 2048 [Fri Apr 12 11:14:12 2019][2937773.892545] .runnable_load_avg : 0 [Fri Apr 12 11:14:12 2019][2937773.897338] .blocked_load_avg : 274 [Fri Apr 12 11:14:12 2019][2937773.902311] .tg_load_avg : 0 [Fri Apr 12 11:14:12 2019][2937773.907104] .tg_load_contrib : 0 [Fri Apr 12 11:14:12 2019][2937773.911898] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:12 2019][2937773.916689] .tg->runnable_avg : 0 [Fri Apr 12 11:14:12 2019][2937773.921483] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:12 2019][2937773.926277] .throttled : 0 [Fri Apr 12 11:14:12 2019][2937773.931070] .throttle_count : 0 [Fri Apr 12 11:14:12 2019][2937773.935886] .avg->runnable_avg_sum : 26648 [Fri Apr 12 11:14:12 2019][2937773.941027] .avg->runnable_avg_period : 46776 [Fri Apr 12 11:14:12 2019][2937773.946173] [Fri Apr 12 11:14:12 2019][2937773.947848] rt_rq[13]:/ [Fri Apr 12 11:14:12 2019][2937773.950473] .rt_nr_running : 0 [Fri Apr 12 11:14:12 2019][2937773.955267] .rt_throttled : 0 [Fri Apr 12 11:14:12 2019][2937773.960058] .rt_time : 0.000000 [Fri Apr 12 11:14:12 2019][2937773.965459] .rt_runtime : 950.000000 [Fri Apr 12 11:14:12 2019][2937773.971032] [Fri Apr 12 11:14:12 2019][2937773.972703] runnable tasks: [Fri Apr 12 11:14:12 2019][2937773.975676] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:12 2019][2937773.986640] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:12 2019][2937773.997519] watchdog/13 76 -11.979110 734429 0 0.000000 57532.947867 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.008863] migration/13 77 0.000000 47453 0 0.000000 634350.580412 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.020198] ksoftirqd/13 78 129171382.751626 43742609 120 0.000000 591956.073622 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.031535] kworker/13:0H 80 3475.303235 9 100 0.000000 0.039386 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.042921] multipathd 14438 0.000000 106 0 0.000000 15.245122 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.054274] kdmflush 17091 3469.077919 2 100 0.000000 0.012714 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.065620] bioset 17104 3469.991885 2 100 0.000000 0.013546 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.076994] kworker/13:1H 17310 129171400.362212 39446191 100 0.000000 242370.243678 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.088328] kdmflush 17325 3487.359439 2 100 0.000000 0.012844 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.099665] bioset 17328 3487.362689 2 100 0.000000 0.012453 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.111021] bioset 17542 3490.505483 2 100 0.000000 0.012274 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.122375] kworker/13:1 85666 129171325.508100 2724377 120 0.000000 73161.548962 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.133727] ptlrpcd_01_00 117146 129170286.172921 648927 120 0.000000 16400.369309 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.145149] ptlrpcd_01_06 117152 129169462.948006 651558 120 0.000000 16488.416696 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.156571] ptlrpcd_01_08 117154 129170140.394913 650368 120 0.000000 16372.958460 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.167999] ldlm_cn01_001 117270 123974234.339362 53222662 120 0.000000 1302557.610778 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.179417] ldlm_cn01_002 117271 129171429.075476 49371372 120 0.000000 1244722.098645 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.190899] mdt01_008 117906 129081713.443626 26312421 120 0.000000 1830112.081507 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.202333] mdt01_024 118195 129166749.157569 24508465 120 0.000000 1667471.449530 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.213759] mdt01_029 118215 121889084.146316 15524027 120 0.000000 791335.930992 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.225181] mdt01_041 118261 114805868.168099 3737750 120 0.000000 132217.879743 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.236600] mdt01_049 118276 127948468.027132 21114593 120 0.000000 1422123.225699 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.248022] mdt01_051 118281 129047311.118335 27144057 120 0.000000 1824648.992020 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.259446] mdt01_052 118285 129171427.972374 26295728 120 0.000000 1765161.807389 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.270872] mdt01_064 118323 129171440.029112 21803057 120 0.000000 1609163.077807 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.282293] mdt01_071 118333 118686659.029059 5922542 120 0.000000 247381.775937 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.293718] mdt01_088 118355 129100633.326995 21377617 120 0.000000 1591719.871309 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.305140] mdt01_099 118391 114534330.929533 1334308 120 0.000000 47871.869177 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.316562] mdt01_104 118406 129152850.963610 21240172 120 0.000000 1462195.701394 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.328016] mdt_io01_024 118750 129147916.768575 2397211 120 0.000000 187766.491237 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.339454] mdt_io01_039 118842 115443803.016678 2681 120 0.000000 60.844738 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.350908] mdt_rdpg01_012 119057 129171428.036315 25692745 120 0.000000 1202330.613845 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.362335] mdt_io01_084 119078 129144805.146138 3045228 120 0.000000 207689.440151 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.373755] mdt_rdpg01_020 119085 129171439.958959 22446061 120 0.000000 1191838.941525 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.385185] mdt_io01_094 119130 115443803.106969 883 120 0.000000 23.121055 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.396648] mdt_rdpg01_036 119152 129171437.521081 25778436 120 0.000000 1304727.481775 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.408089] mdt_rdpg01_053 119216 129171428.036295 23113890 120 0.000000 1327716.445501 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.419517] mdt_io01_111 119232 129165604.285950 2286303 120 0.000000 181810.744829 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.430952] ldlm_cn01_019 123807 129171451.680908 50835341 120 0.000000 1274338.136955 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.442388] ldlm_cn01_030 129952 129171429.372805 42958710 120 0.000000 1131981.890249 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.453841] kworker/13:0 29284 122450095.982467 3 120 0.000000 0.016852 0.000000 1 / [Fri Apr 12 11:14:12 2019][2937774.465223] [Fri Apr 12 11:14:12 2019][2937774.466901] cpu#14, 1996.203 MHz [Fri Apr 12 11:14:12 2019][2937774.470309] .nr_running : 1 [Fri Apr 12 11:14:12 2019][2937774.475119] .load : 1024 [Fri Apr 12 11:14:12 2019][2937774.480190] .nr_switches : 19916791464 [Fri Apr 12 11:14:12 2019][2937774.485867] .nr_load_updates : 2004915881 [Fri Apr 12 11:14:12 2019][2937774.491457] .nr_uninterruptible : -44355 [Fri Apr 12 11:14:12 2019][2937774.496727] .next_balance : 7232.375244 [Fri Apr 12 11:14:12 2019][2937774.502413] .curr->pid : 121492 [Fri Apr 12 11:14:12 2019][2937774.507674] .clock : 2937774507.576693 [Fri Apr 12 11:14:12 2019][2937774.513859] .cpu_load[0] : 1024 [Fri Apr 12 11:14:12 2019][2937774.518920] .cpu_load[1] : 6075 [Fri Apr 12 11:14:12 2019][2937774.523982] .cpu_load[2] : 4181 [Fri Apr 12 11:14:12 2019][2937774.529043] .cpu_load[3] : 3354 [Fri Apr 12 11:14:12 2019][2937774.534105] .cpu_load[4] : 2564 [Fri Apr 12 11:14:12 2019][2937774.539166] .avg_idle : 360713 [Fri Apr 12 11:14:12 2019][2937774.544403] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:12 2019][2937774.549721] [Fri Apr 12 11:14:12 2019][2937774.551391] cfs_rq[14]:/ [Fri Apr 12 11:14:12 2019][2937774.554105] .exec_clock : 0.000000 [Fri Apr 12 11:14:12 2019][2937774.559520] .MIN_vruntime : 106499167.005985 [Fri Apr 12 11:14:12 2019][2937774.565624] .min_vruntime : 106499169.811469 [Fri Apr 12 11:14:12 2019][2937774.571723] .max_vruntime : 106499167.005985 [Fri Apr 12 11:14:12 2019][2937774.577825] .spread : 0.000000 [Fri Apr 12 11:14:12 2019][2937774.583226] .spread0 : -10343189.580865 [Fri Apr 12 11:14:12 2019][2937774.589318] .nr_spread_over : 0 [Fri Apr 12 11:14:12 2019][2937774.594119] .nr_running : 0 [Fri Apr 12 11:14:12 2019][2937774.598912] .load : 1024 [Fri Apr 12 11:14:12 2019][2937774.603965] .runnable_load_avg : 0 [Fri Apr 12 11:14:12 2019][2937774.608758] .blocked_load_avg : 1157 [Fri Apr 12 11:14:12 2019][2937774.613811] .tg_load_avg : 0 [Fri Apr 12 11:14:12 2019][2937774.618602] .tg_load_contrib : 0 [Fri Apr 12 11:14:12 2019][2937774.623395] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:12 2019][2937774.628187] .tg->runnable_avg : 0 [Fri Apr 12 11:14:12 2019][2937774.632989] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:12 2019][2937774.637781] .throttled : 0 [Fri Apr 12 11:14:12 2019][2937774.642576] .throttle_count : 0 [Fri Apr 12 11:14:12 2019][2937774.647385] .avg->runnable_avg_sum : 12899 [Fri Apr 12 11:14:12 2019][2937774.652542] .avg->runnable_avg_period : 46219 [Fri Apr 12 11:14:12 2019][2937774.657690] [Fri Apr 12 11:14:12 2019][2937774.659371] rt_rq[14]:/ [Fri Apr 12 11:14:12 2019][2937774.661996] .rt_nr_running : 0 [Fri Apr 12 11:14:12 2019][2937774.666798] .rt_throttled : 0 [Fri Apr 12 11:14:12 2019][2937774.671592] .rt_time : 0.000000 [Fri Apr 12 11:14:12 2019][2937774.676992] .rt_runtime : 950.000000 [Fri Apr 12 11:14:12 2019][2937774.682564] [Fri Apr 12 11:14:12 2019][2937774.684236] runnable tasks: [Fri Apr 12 11:14:12 2019][2937774.687208] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:12 2019][2937774.698156] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:12 2019][2937774.709043] watchdog/14 81 -11.979091 734378 0 0.000000 40488.710998 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.720403] migration/14 82 0.000000 32724 0 0.000000 635334.107853 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.731739] ksoftirqd/14 83 106499165.937231 40488055 120 0.000000 730157.491106 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.743092] kworker/14:0H 85 4451.558204 10 100 0.000000 0.032092 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.754456] scsi_eh_1 5579 1129.976506 2 120 0.000000 0.009398 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.765802] kworker/14:1 7296 106499100.804850 2668940 120 0.000000 66372.853523 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.777144] ccp-2-q2 13500 3362.374470 2 120 0.000000 0.013636 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.788482] ccp-2-q4 13504 3370.386351 2 120 0.000000 0.011883 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.799818] ccp-4-q4 13548 3384.618674 2 120 0.000000 0.013045 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.811189] writer#1 19292 106498059.392075 15160438 120 0.000000 236451.800528 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.822522] writer#2 19293 106498059.333274 15221525 120 0.000000 236052.093841 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.833862] reader#0 19296 106498059.565674 3415751 120 0.000000 791953.849652 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.845202] slapd 19733 105791286.505976 2431 120 0.000000 2440.847619 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.856536] kworker/14:1H 20662 106499180.082942 23701932 100 0.000000 139059.840408 0.000000 2 / [Fri Apr 12 11:14:12 2019][2937774.867883] ptlrpc_hr02_003 117089 106499103.855069 2388801 120 0.000000 75010.055522 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.879326] ptlrpcd_02_02 117160 106498754.970543 834625 120 0.000000 22097.893808 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.890758] ptlrpcd_02_06 117164 106498906.596627 832790 120 0.000000 21975.664372 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.902181] ptlrpcd_02_08 117166 106498956.383376 835520 120 0.000000 22185.159640 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.913609] ldlm_cn02_001 117273 106499157.446905 43959711 120 0.000000 1067772.893483 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.925036] mdt_out02_000 117496 106494656.890813 86026 120 0.000000 2964.489398 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.936465] mdt02_005 117533 106498947.036391 11140615 120 0.000000 986974.276488 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.947894] osp-syn-46-2 117574 106498512.086035 236045 120 0.000000 6562.926756 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.959323] osp-syn-40-2 117626 106498701.390535 236564 120 0.000000 6871.787541 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.970776] ldlm_cn02_006 117901 106499157.447607 46114722 120 0.000000 1111224.770662 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.982203] ll_mgs_0007 117905 106499042.818850 2981835 120 0.000000 101540.230263 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937774.993634] mdt02_014 118184 106499206.734873 30195203 120 0.000000 1825932.334789 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.005067] mdt02_032 118284 100655279.221901 8734919 120 0.000000 577429.454112 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.016491] mdt02_037 118307 106470476.920642 19739435 120 0.000000 1286977.564055 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.027917] mdt02_042 118359 103163052.081036 13290955 120 0.000000 839938.975452 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.039342] mdt02_046 118367 106499213.421083 19818104 120 0.000000 1383150.561845 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.050767] mdt02_050 118392 93048053.644404 1260368 120 0.000000 50302.029266 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.062191] mdt_io02_003 118413 106476633.497704 325368 120 0.000000 24949.542675 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.073612] mdt02_054 118417 94407048.294610 1262566 120 0.000000 53942.842111 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.085037] mdt_io02_004 118433 94400416.311224 106 120 0.000000 4.479590 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.096457] mdt02_056 118438 99836621.065752 1545307 120 0.000000 161000.574799 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.107883] mdt02_067 118475 106499237.237047 17187299 120 0.000000 1186087.558138 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.119312] mdt02_097 118565 106499200.096396 20229471 120 0.000000 1402211.467717 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.130734] mdt02_098 118568 106425414.462512 14342503 120 0.000000 948776.637089 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.142169] mdt_io02_026 118677 106002193.519029 1786259 120 0.000000 110262.785087 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.153595] mdt_io02_042 118733 106499226.471634 2043818 120 0.000000 190008.831345 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.165023] mdt_io02_049 118758 94400419.905699 6422 120 0.000000 270.688684 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.176459] mdt_io02_074 118862 106451903.163319 1986392 120 0.000000 170089.258591 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.187890] mdt_io02_082 118896 106496487.314428 203748 120 0.000000 15302.246736 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.199316] mdt_io02_100 118956 106499232.484236 2297652 120 0.000000 216874.374177 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.210743] mdt_io02_108 118996 106451903.162087 3086566 120 0.000000 228799.515077 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.222167] mdt_rdpg02_009 119027 106471367.260645 21348171 120 0.000000 1071178.789262 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.233589] mdt_rdpg02_011 119046 106499240.248944 19410638 120 0.000000 1002124.508585 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.245010] mdt_rdpg02_013 119054 106495050.690534 21370593 120 0.000000 1015391.399602 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.256437] mdt_rdpg02_027 119103 106495044.069805 13827277 120 0.000000 767279.798229 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.267865] mdt_rdpg02_028 119119 106499221.983328 15235115 120 0.000000 918858.730584 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.279301] mdt_rdpg02_055 119234 106472557.078997 15791904 120 0.000000 820205.811220 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.290743] ldlm_bl_09 128942 106499237.755372 702333 120 0.000000 90564.040167 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.302167] ldlm_cn02_021 129946 106499230.131439 33600809 120 0.000000 867150.010505 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.313600] osp-pre-6-3 2295 106498884.759434 28034 120 0.000000 619.360222 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.324939] osp-pre-18-1 2419 106498787.187482 27803 120 0.000000 606.943664 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.336286] ldlm_cn02_023 6478 106499222.028123 30379146 120 0.000000 788288.011214 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.347624] ldlm_cn02_026 6481 106499042.952292 30050480 120 0.000000 791188.513645 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.358961] ldlm_cn02_030 6496 106499237.127069 33479232 120 0.000000 850355.080007 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.370296] ldlm_cn02_031 6497 106499227.089367 32795070 120 0.000000 833551.565943 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.381633] kworker/14:0 6950 94885547.395940 2 120 0.000000 0.011472 0.000000 2 / [Fri Apr 12 11:14:13 2019][2937775.393018] [Fri Apr 12 11:14:13 2019][2937775.394696] cpu#15, 1996.203 MHz [Fri Apr 12 11:14:13 2019][2937775.398104] .nr_running : 5 [Fri Apr 12 11:14:13 2019][2937775.402896] .load : 3072 [Fri Apr 12 11:14:13 2019][2937775.407949] .nr_switches : 11581725754 [Fri Apr 12 11:14:13 2019][2937775.413609] .nr_load_updates : 1823641902 [Fri Apr 12 11:14:13 2019][2937775.419181] .nr_uninterruptible : -27632 [Fri Apr 12 11:14:13 2019][2937775.424407] .next_balance : 7232.376265 [Fri Apr 12 11:14:13 2019][2937775.430067] .curr->pid : 46544 [Fri Apr 12 11:14:13 2019][2937775.435221] .clock : 2937535117.389000 [Fri Apr 12 11:14:13 2019][2937775.441403] .cpu_load[0] : 3072 [Fri Apr 12 11:14:13 2019][2937775.446454] .cpu_load[1] : 3072 [Fri Apr 12 11:14:13 2019][2937775.451508] .cpu_load[2] : 3072 [Fri Apr 12 11:14:13 2019][2937775.456561] .cpu_load[3] : 3072 [Fri Apr 12 11:14:13 2019][2937775.461622] .cpu_load[4] : 3072 [Fri Apr 12 11:14:13 2019][2937775.466675] .avg_idle : 234361 [Fri Apr 12 11:14:13 2019][2937775.471902] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:13 2019][2937775.477215] [Fri Apr 12 11:14:13 2019][2937775.478883] cfs_rq[15]:/ [Fri Apr 12 11:14:13 2019][2937775.481596] .exec_clock : 0.000000 [Fri Apr 12 11:14:13 2019][2937775.486998] .MIN_vruntime : 81551159.722118 [Fri Apr 12 11:14:13 2019][2937775.493001] .min_vruntime : 81551171.722118 [Fri Apr 12 11:14:13 2019][2937775.499007] .max_vruntime : 81551159.722118 [Fri Apr 12 11:14:13 2019][2937775.505014] .spread : 0.000000 [Fri Apr 12 11:14:13 2019][2937775.510413] .spread0 : -35291271.502030 [Fri Apr 12 11:14:13 2019][2937775.516507] .nr_spread_over : 0 [Fri Apr 12 11:14:13 2019][2937775.521298] .nr_running : 3 [Fri Apr 12 11:14:13 2019][2937775.526092] .load : 3072 [Fri Apr 12 11:14:13 2019][2937775.531145] .runnable_load_avg : 179 [Fri Apr 12 11:14:13 2019][2937775.536110] .blocked_load_avg : 0 [Fri Apr 12 11:14:13 2019][2937775.540905] .tg_load_avg : 0 [Fri Apr 12 11:14:13 2019][2937775.545697] .tg_load_contrib : 0 [Fri Apr 12 11:14:13 2019][2937775.550489] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:13 2019][2937775.555282] .tg->runnable_avg : 0 [Fri Apr 12 11:14:13 2019][2937775.560075] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:13 2019][2937775.564867] .throttled : 0 [Fri Apr 12 11:14:13 2019][2937775.569662] .throttle_count : 0 [Fri Apr 12 11:14:13 2019][2937775.574454] .avg->runnable_avg_sum : 17376 [Fri Apr 12 11:14:13 2019][2937775.579594] .avg->runnable_avg_period : 46522 [Fri Apr 12 11:14:13 2019][2937775.584741] [Fri Apr 12 11:14:13 2019][2937775.586425] rt_rq[15]:/ [Fri Apr 12 11:14:13 2019][2937775.589049] .rt_nr_running : 1 [Fri Apr 12 11:14:13 2019][2937775.593840] .rt_throttled : 0 [Fri Apr 12 11:14:13 2019][2937775.598632] .rt_time : 0.000000 [Fri Apr 12 11:14:13 2019][2937775.604033] .rt_runtime : 1000.000000 [Fri Apr 12 11:14:13 2019][2937775.609693] [Fri Apr 12 11:14:13 2019][2937775.611364] runnable tasks: [Fri Apr 12 11:14:13 2019][2937775.614338] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:13 2019][2937775.625282] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:13 2019][2937775.636153] watchdog/15 86 -11.980694 734371 0 0.000000 22835.316913 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.647486] migration/15 87 0.000000 8136 0 0.000000 71874.577137 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.658823] ksoftirqd/15 88 81551159.722118 15710236 120 0.000000 171210.792860 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.670159] kworker/15:0H 90 1559.193787 9 100 0.000000 0.032533 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.681528] kworker/15:1H 7448 81551050.547039 14111691 100 0.000000 72216.071104 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.692897] kworker/15:1 8937 81551159.722118 1384552 120 0.000000 27562.146439 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.704250] ldlm_cb03_001 117265 81501797.803383 1014 120 0.000000 25.540324 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.715684] mdt03_000 117472 81241768.105855 1485277 120 0.000000 81304.549641 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.727110] mdt03_002 117474 60177004.675879 970065 120 0.000000 32846.339451 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.738540] mdt_seqm_0000 117502 81526174.763549 75 120 0.000000 2.628640 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.750013] osp-syn-34-0 117814 81530371.493887 75281 120 0.000000 2219.616919 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.761445] mdt03_012 118203 81241675.257765 16159595 120 0.000000 932695.216259 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.772872] mdt03_013 118222 60172349.056547 394279 120 0.000000 13214.011984 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.784329] mdt_io03_004 118598 64277445.217616 48 120 0.000000 1.690778 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.795762] mdt03_077 118664 66302973.520686 1190438 120 0.000000 50780.902405 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.807185] mdt_io03_018 118675 64277445.292608 142 120 0.000000 6.019349 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.818609] mdt_io03_023 118697 64277445.265788 519526 120 0.000000 21657.193352 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.830031] mdt_io03_024 118705 64277467.030459 74 120 0.000000 3.026276 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.841456] mdt03_095 118738 81241768.106136 9507720 120 0.000000 429081.764454 0.000000 3 / [Fri Apr 12 11:14:13 2019][2937775.852890] mdt_io03_058 118839 64277445.105472 65 120 0.000000 0.983980 0.000000 3 / [Fri Apr 12 11:14:14 2019][2937775.864318] mdt_io03_063 118851 64277445.255627 46 120 0.000000 1.813365 0.000000 3 / [Fri Apr 12 11:14:14 2019][2937775.875740] mdt_io03_065 118863 64277445.308107 55 120 0.000000 2.135952 0.000000 3 / [Fri Apr 12 11:14:14 2019][2937775.887236] mdt_out03_002 119288 81528205.254365 182436 120 0.000000 6068.637397 0.000000 3 / [Fri Apr 12 11:14:14 2019][2937775.898704] kworker/15:2 35529 77752358.212093 2 120 0.000000 0.007495 0.000000 3 / [Fri Apr 12 11:14:14 2019][2937775.910049] R bash 46544 81551169.287238 88 120 0.000000 28.487281 0.000000 3 / [Fri Apr 12 11:14:14 2019][2937775.921399] [Fri Apr 12 11:14:14 2019][2937775.923071] cpu#16, 1996.203 MHz [Fri Apr 12 11:14:14 2019][2937775.926480] .nr_running : 1 [Fri Apr 12 11:14:14 2019][2937775.931283] .load : 1024 [Fri Apr 12 11:14:14 2019][2937775.936341] .nr_switches : 19588446095 [Fri Apr 12 11:14:14 2019][2937775.942004] .nr_load_updates : 2132469133 [Fri Apr 12 11:14:14 2019][2937775.947591] .nr_uninterruptible : 622012 [Fri Apr 12 11:14:14 2019][2937775.952818] .next_balance : 7232.376700 [Fri Apr 12 11:14:14 2019][2937775.958487] .curr->pid : 0 [Fri Apr 12 11:14:14 2019][2937775.963296] .clock : 2937775963.281328 [Fri Apr 12 11:14:14 2019][2937775.969476] .cpu_load[0] : 0 [Fri Apr 12 11:14:14 2019][2937775.974278] .cpu_load[1] : 12 [Fri Apr 12 11:14:14 2019][2937775.979166] .cpu_load[2] : 464 [Fri Apr 12 11:14:14 2019][2937775.984141] .cpu_load[3] : 459 [Fri Apr 12 11:14:14 2019][2937775.989115] .cpu_load[4] : 288 [Fri Apr 12 11:14:14 2019][2937775.994108] .avg_idle : 118291 [Fri Apr 12 11:14:14 2019][2937775.999341] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:14 2019][2937776.004656] [Fri Apr 12 11:14:14 2019][2937776.006333] cfs_rq[16]:/ [Fri Apr 12 11:14:14 2019][2937776.009044] .exec_clock : 0.000000 [Fri Apr 12 11:14:14 2019][2937776.014445] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:14 2019][2937776.019853] .min_vruntime : 119337956.566707 [Fri Apr 12 11:14:14 2019][2937776.025953] .max_vruntime : 0.000001 [Fri Apr 12 11:14:14 2019][2937776.031354] .spread : 0.000000 [Fri Apr 12 11:14:14 2019][2937776.036751] .spread0 : 2495468.399817 [Fri Apr 12 11:14:14 2019][2937776.042672] .nr_spread_over : 0 [Fri Apr 12 11:14:14 2019][2937776.047466] .nr_running : 0 [Fri Apr 12 11:14:14 2019][2937776.052258] .load : 0 [Fri Apr 12 11:14:14 2019][2937776.057052] .runnable_load_avg : 0 [Fri Apr 12 11:14:14 2019][2937776.061844] .blocked_load_avg : 600 [Fri Apr 12 11:14:14 2019][2937776.066809] .tg_load_avg : 0 [Fri Apr 12 11:14:14 2019][2937776.071603] .tg_load_contrib : 0 [Fri Apr 12 11:14:14 2019][2937776.076412] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:14 2019][2937776.081205] .tg->runnable_avg : 0 [Fri Apr 12 11:14:14 2019][2937776.085998] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:14 2019][2937776.090789] .throttled : 0 [Fri Apr 12 11:14:14 2019][2937776.095584] .throttle_count : 0 [Fri Apr 12 11:14:14 2019][2937776.100378] .avg->runnable_avg_sum : 10797 [Fri Apr 12 11:14:14 2019][2937776.105516] .avg->runnable_avg_period : 46141 [Fri Apr 12 11:14:14 2019][2937776.110663] [Fri Apr 12 11:14:14 2019][2937776.112337] rt_rq[16]:/ [Fri Apr 12 11:14:14 2019][2937776.114961] .rt_nr_running : 0 [Fri Apr 12 11:14:14 2019][2937776.119755] .rt_throttled : 0 [Fri Apr 12 11:14:14 2019][2937776.124547] .rt_time : 0.000000 [Fri Apr 12 11:14:14 2019][2937776.129947] .rt_runtime : 950.000000 [Fri Apr 12 11:14:14 2019][2937776.135519] [Fri Apr 12 11:14:14 2019][2937776.137193] runnable tasks: [Fri Apr 12 11:14:14 2019][2937776.140165] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:14 2019][2937776.151112] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:14 2019][2937776.161981] watchdog/16 91 -11.984411 734430 0 0.000000 38368.528929 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.173314] migration/16 92 0.000000 27661 0 0.000000 353102.736926 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.184652] ksoftirqd/16 93 119337952.145744 25479820 120 0.000000 366355.548247 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.195997] kworker/16:0H 95 4579.574903 9 100 0.000000 0.034755 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.207385] kworker/16:1H 16828 119337952.125476 22723701 100 0.000000 148743.469984 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.218741] smartd 18024 118951211.845703 18147 120 0.000000 641.732654 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.230077] dsm_sa_eventmgr 19169 119335701.720625 492612 120 0.000000 26491.202814 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.241434] kworker/16:0 109801 119337954.687023 1018909 120 0.000000 18062.034884 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.252872] ptlrpcd_00_04 117138 119337944.461342 855621 120 0.000000 23463.232110 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.264292] ptlrpcd_00_11 117145 119337799.608171 879422 120 0.000000 22536.257607 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.275721] ldlm_cn00_001 117267 119337962.505777 42127448 120 0.000000 998928.906289 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.287149] mgs_fir_notify 117298 103123759.182613 17 118 0.000000 84.023146 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.298578] jbd2/dm-3-8 117447 119337741.070714 401330 120 0.000000 2521300.784947 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.310002] mdt00_000 117463 102880373.230410 4909580 120 0.000000 181678.925562 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.321429] mdt_out00_001 117493 119334205.492723 13105504 120 0.000000 336687.238045 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.332871] mdt00_005 117551 119241125.155407 36508782 120 0.000000 1707936.935979 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.344311] osp-syn-2-2 117590 119337799.614292 236647 120 0.000000 6648.296893 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.355748] osp-syn-44-2 117606 119337799.667863 235769 120 0.000000 6648.664891 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.367209] mdt00_008 117969 119334687.309072 34980765 120 0.000000 1747676.417464 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.378647] mdt00_010 118157 119222222.913349 28425206 120 0.000000 1660086.826644 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.390065] mdt00_011 118158 119334693.166802 24322920 120 0.000000 1254838.042248 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.401490] mdt00_015 118172 119333305.422502 32097158 120 0.000000 1695631.876181 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.412925] mdt00_049 118280 119220269.415927 30792815 120 0.000000 1727117.449507 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.424351] mdt00_055 118299 119330700.496567 29462312 120 0.000000 1495253.247870 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.435777] mdt00_061 118318 119330700.498141 29068629 120 0.000000 1657498.560857 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.447212] mdt00_064 118369 113687981.207174 19153699 120 0.000000 871376.754973 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.458642] mdt00_075 118421 119319639.074810 23351847 120 0.000000 1417266.198531 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.470067] mdt_rdpg00_005 118493 119335146.763379 26030888 120 0.000000 1142256.303145 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.481502] mdt_rdpg00_008 118497 119337471.628945 23853907 120 0.000000 1051313.280928 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.492944] mdt00_093 118536 119337437.302952 43453905 120 0.000000 2039410.272107 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.504392] mdt00_104 118559 119334686.885139 14288888 120 0.000000 926679.773805 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.515822] mdt00_107 118564 119216658.805210 19633096 120 0.000000 1212197.228911 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.527245] mdt00_110 118572 119333305.514777 21945091 120 0.000000 1447102.108910 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.538670] mdt_io00_008 118604 103125050.064267 88 120 0.000000 4.443287 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.550096] mdt_io00_018 118635 103125375.728196 64 120 0.000000 3.978209 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.561530] mdt_io00_026 118671 103125050.345181 523752 120 0.000000 21058.450146 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.572971] mdt_io00_053 118836 103125049.697230 59 120 0.000000 3.403711 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.584408] mdt_io00_056 118854 103125049.716817 465588 120 0.000000 19739.020173 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.595841] mdt_io00_066 118893 103125049.763736 53 120 0.000000 2.351475 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.607269] mdt_io00_069 118909 103125049.774076 66 120 0.000000 4.674836 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.618707] mdt_io00_083 118971 103125049.712830 58 120 0.000000 2.262950 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.630150] mdt_io00_084 118974 119298307.673660 311 120 0.000000 5.937680 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.641588] mdt_io00_085 118980 103125049.611166 42 120 0.000000 1.288043 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.653050] mdt_rdpg00_012 119032 119331517.011803 15298824 120 0.000000 701537.831420 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.664487] mdt_rdpg00_015 119041 119337471.625319 15152896 120 0.000000 852932.749225 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.675910] mdt_rdpg00_017 119045 119337978.770160 22852043 120 0.000000 1067451.782338 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.687339] mdt_io00_106 119090 103125050.143555 35 120 0.000000 1.834823 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.698770] mdt_rdpg00_027 119126 119335146.763069 20594872 120 0.000000 1007317.206119 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.710197] mdt_rdpg00_031 119148 119331517.011532 21705131 120 0.000000 987867.444293 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.721637] mdt_rdpg00_049 119227 119337986.551619 18790446 120 0.000000 859878.152505 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.733065] ldlm_cn00_009 120497 119337619.174606 45497277 120 0.000000 1067329.682332 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.744511] ldlm_cn00_014 126370 119337630.675831 34404272 120 0.000000 857886.447104 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.755950] ldlm_cn00_018 127575 119337627.382605 39279074 120 0.000000 953609.618015 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.767380] osp-pre-9-3 2253 119337595.126580 28020 120 0.000000 632.837610 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.778716] osp-pre-34-3 2327 119337772.107363 27750 120 0.000000 736.613109 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.790071] ldlm_cn00_028 6485 119337619.173854 33216489 120 0.000000 811680.078221 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.801411] ldlm_cn00_030 6487 119338000.919696 30538525 120 0.000000 766737.818129 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.812750] mdt_out00_004 8935 119334288.648042 8649463 120 0.000000 215757.611783 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.824092] kworker/16:1 40977 118165202.146013 4 120 0.000000 0.013757 0.000000 0 / [Fri Apr 12 11:14:14 2019][2937776.835451] [Fri Apr 12 11:14:14 2019][2937776.837123] cpu#17, 1996.203 MHz [Fri Apr 12 11:14:14 2019][2937776.840532] .nr_running : 2 [Fri Apr 12 11:14:14 2019][2937776.845335] .load : 0 [Fri Apr 12 11:14:14 2019][2937776.850134] .nr_switches : 20352933037 [Fri Apr 12 11:14:14 2019][2937776.855811] .nr_load_updates : 2185301456 [Fri Apr 12 11:14:14 2019][2937776.861395] .nr_uninterruptible : -78898 [Fri Apr 12 11:14:15 2019][2937776.866639] .next_balance : 7232.377611 [Fri Apr 12 11:14:15 2019][2937776.872305] .curr->pid : 119110 [Fri Apr 12 11:14:15 2019][2937776.877539] .clock : 2937776877.537123 [Fri Apr 12 11:14:15 2019][2937776.883728] .cpu_load[0] : 0 [Fri Apr 12 11:14:15 2019][2937776.888538] .cpu_load[1] : 1153 [Fri Apr 12 11:14:15 2019][2937776.893600] .cpu_load[2] : 2154 [Fri Apr 12 11:14:15 2019][2937776.898659] .cpu_load[3] : 1342 [Fri Apr 12 11:14:15 2019][2937776.903711] .cpu_load[4] : 1249 [Fri Apr 12 11:14:15 2019][2937776.908763] .avg_idle : 39459 [Fri Apr 12 11:14:15 2019][2937776.913904] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:15 2019][2937776.919235] [Fri Apr 12 11:14:15 2019][2937776.920911] cfs_rq[17]:/ [Fri Apr 12 11:14:15 2019][2937776.923625] .exec_clock : 0.000000 [Fri Apr 12 11:14:15 2019][2937776.929025] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:15 2019][2937776.934423] .min_vruntime : 131266004.998692 [Fri Apr 12 11:14:15 2019][2937776.940516] .max_vruntime : 0.000001 [Fri Apr 12 11:14:15 2019][2937776.945915] .spread : 0.000000 [Fri Apr 12 11:14:15 2019][2937776.951315] .spread0 : 14423451.827164 [Fri Apr 12 11:14:15 2019][2937776.957321] .nr_spread_over : 0 [Fri Apr 12 11:14:15 2019][2937776.962113] .nr_running : 3 [Fri Apr 12 11:14:15 2019][2937776.966905] .load : 0 [Fri Apr 12 11:14:15 2019][2937776.971699] .runnable_load_avg : 488 [Fri Apr 12 11:14:15 2019][2937776.976665] .blocked_load_avg : 1334 [Fri Apr 12 11:14:15 2019][2937776.981717] .tg_load_avg : 0 [Fri Apr 12 11:14:15 2019][2937776.986512] .tg_load_contrib : 0 [Fri Apr 12 11:14:15 2019][2937776.991304] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:15 2019][2937776.996097] .tg->runnable_avg : 0 [Fri Apr 12 11:14:15 2019][2937777.000888] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:15 2019][2937777.005682] .throttled : 0 [Fri Apr 12 11:14:15 2019][2937777.010474] .throttle_count : 0 [Fri Apr 12 11:14:15 2019][2937777.015267] .avg->runnable_avg_sum : 24763 [Fri Apr 12 11:14:15 2019][2937777.020406] .avg->runnable_avg_period : 46465 [Fri Apr 12 11:14:15 2019][2937777.025554] [Fri Apr 12 11:14:15 2019][2937777.027228] rt_rq[17]:/ [Fri Apr 12 11:14:15 2019][2937777.029854] .rt_nr_running : 0 [Fri Apr 12 11:14:15 2019][2937777.034645] .rt_throttled : 0 [Fri Apr 12 11:14:15 2019][2937777.039439] .rt_time : 0.000000 [Fri Apr 12 11:14:15 2019][2937777.044840] .rt_runtime : 950.000000 [Fri Apr 12 11:14:15 2019][2937777.050412] [Fri Apr 12 11:14:15 2019][2937777.052082] runnable tasks: [Fri Apr 12 11:14:15 2019][2937777.055055] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:15 2019][2937777.066001] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:15 2019][2937777.076875] watchdog/17 96 -11.979902 734430 0 0.000000 56898.825646 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.088225] migration/17 97 0.000000 46702 0 0.000000 610339.016439 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.099561] ksoftirqd/17 98 131265985.722110 35660924 120 0.000000 539180.364005 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.110906] kworker/17:0H 100 3784.833540 9 100 0.000000 0.029406 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.122262] kdevtmpfs 255 113918780.197038 320 120 0.000000 3.280075 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.133626] ccp-2-q0 13496 3071.812176 2 120 0.000000 0.023304 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.144960] ccp-2-q3 13502 3079.829467 2 120 0.000000 0.012364 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.156326] kworker/17:1H 17602 131265985.711453 30687222 100 0.000000 202726.542382 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.167697] ptlrpc_hr01_001 117081 131266021.192136 3117455 120 0.000000 110230.913400 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.179123] ptlrpc_hr01_002 117082 131266042.901043 3118070 120 0.000000 109622.752239 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.190545] ptlrpc_hr01_003 117083 131266050.394926 3117547 120 0.000000 109508.040876 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.201983] ldlm_elt 117280 126755141.513303 24 120 0.000000 351.757438 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.213421] mdt01_004 117534 131266067.799399 31917182 120 0.000000 1956069.679135 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.224842] mdt01_005 117552 131266067.600788 26825869 120 0.000000 1747102.948124 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.236262] mdt01_006 117557 114072286.641002 815113 120 0.000000 25392.902826 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.247718] ldlm_cn01_005 117687 131773884.551707 51688132 120 0.000000 1294966.680694 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.259172] mdt01_012 118152 131180551.308411 26554170 120 0.000000 1739485.250067 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.270733] mdt01_014 118156 115518675.793412 3001884 120 0.000000 114478.542960 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.282184] mdt01_015 118163 114063679.784680 1130792 120 0.000000 35108.751724 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.293615] mdt01_030 118219 131168743.556270 21940938 120 0.000000 1581440.976525 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.305059] mdt01_053 118290 131241415.928169 23380854 120 0.000000 1554195.602804 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.316485] mdt01_054 118296 131266095.514087 25316018 120 0.000000 1729585.884851 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.327909] mdt01_064 118323 131266103.921381 21804916 120 0.000000 1609464.908179 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.339329] mdt01_066 118326 131190770.204506 26233701 120 0.000000 1753547.942829 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.350755] mdt01_077 118340 122889992.206904 6157096 120 0.000000 435582.705692 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.362179] mdt01_091 118368 126264721.881853 16942632 120 0.000000 1074657.316855 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.373599] mdt01_093 118371 114422589.795558 1710400 120 0.000000 60682.127613 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.385023] mdt01_095 118382 131166627.881999 23825384 120 0.000000 1603801.803828 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.396448] mdt01_105 118410 131180446.069577 20413853 120 0.000000 1476055.853168 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.407885] mdt_io01_010 118613 121832996.905648 71106 120 0.000000 3503.109536 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.419324] mdt_io01_023 118746 116328142.243866 415407 120 0.000000 18796.314727 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.430793] mdt_io01_073 119039 116328143.118474 77 120 0.000000 3.678118 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.442230] mdt_rdpg01_035 119142 131266121.470755 22755367 120 0.000000 1182020.617042 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.453668] mdt_rdpg01_059 119245 131266065.222027 30401865 120 0.000000 1426293.375610 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.465090] ldlm_bl_04 119549 131266005.550492 1063071 120 0.000000 51699.193638 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.476551] kworker/17:0 24866 131266073.872243 102509 120 0.000000 9112.469382 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.487893] kworker/17:2 43376 129831551.553423 2 120 0.000000 0.010209 0.000000 1 / [Fri Apr 12 11:14:15 2019][2937777.499242] [Fri Apr 12 11:14:15 2019][2937777.500917] cpu#18, 1996.203 MHz [Fri Apr 12 11:14:15 2019][2937777.504324] .nr_running : 0 [Fri Apr 12 11:14:15 2019][2937777.509117] .load : 1024 [Fri Apr 12 11:14:15 2019][2937777.514171] .nr_switches : 13427601977 [Fri Apr 12 11:14:15 2019][2937777.519830] .nr_load_updates : 1909525528 [Fri Apr 12 11:14:15 2019][2937777.525403] .nr_uninterruptible : -65881 [Fri Apr 12 11:14:15 2019][2937777.530629] .next_balance : 7232.378400 [Fri Apr 12 11:14:15 2019][2937777.536289] .curr->pid : 118188 [Fri Apr 12 11:14:15 2019][2937777.541514] .clock : 2937777541.430544 [Fri Apr 12 11:14:15 2019][2937777.547695] .cpu_load[0] : 0 [Fri Apr 12 11:14:15 2019][2937777.552514] .cpu_load[1] : 217 [Fri Apr 12 11:14:15 2019][2937777.557489] .cpu_load[2] : 90 [Fri Apr 12 11:14:15 2019][2937777.562375] .cpu_load[3] : 105 [Fri Apr 12 11:14:15 2019][2937777.567342] .cpu_load[4] : 150 [Fri Apr 12 11:14:15 2019][2937777.572308] .avg_idle : 79303 [Fri Apr 12 11:14:15 2019][2937777.577448] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:15 2019][2937777.582768] [Fri Apr 12 11:14:15 2019][2937777.584439] cfs_rq[18]:/ [Fri Apr 12 11:14:15 2019][2937777.587151] .exec_clock : 0.000000 [Fri Apr 12 11:14:15 2019][2937777.592549] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:15 2019][2937777.597950] .min_vruntime : 69694802.896455 [Fri Apr 12 11:14:15 2019][2937777.603957] .max_vruntime : 0.000001 [Fri Apr 12 11:14:15 2019][2937777.609356] .spread : 0.000000 [Fri Apr 12 11:14:15 2019][2937777.614756] .spread0 : -47147872.181923 [Fri Apr 12 11:14:15 2019][2937777.620847] .nr_spread_over : 0 [Fri Apr 12 11:14:15 2019][2937777.625639] .nr_running : 0 [Fri Apr 12 11:14:15 2019][2937777.630433] .load : 0 [Fri Apr 12 11:14:15 2019][2937777.635225] .runnable_load_avg : 0 [Fri Apr 12 11:14:15 2019][2937777.640020] .blocked_load_avg : 687 [Fri Apr 12 11:14:15 2019][2937777.644984] .tg_load_avg : 0 [Fri Apr 12 11:14:15 2019][2937777.649777] .tg_load_contrib : 0 [Fri Apr 12 11:14:15 2019][2937777.654571] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:15 2019][2937777.659363] fs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:15 2019][2937777.668948] .throttled : 0 [Fri Apr 12 11:14:15 2019][2937777.673742] .throttle_count : 0 [Fri Apr 12 11:14:15 2019][2937777.678533] .avg->runnable_avg_sum : 6853 [Fri Apr 12 11:14:15 2019][2937777.683587] .avg->runnable_avg_period : 46500 [Fri Apr 12 11:14:15 2019][2937777.688733] [Fri Apr 12 11:14:15 2019][2937777.690407] rt_rq[18]:/ [Fri Apr 12 11:14:15 2019][2937777.693033] .rt_nr_running : 0 [Fri Apr 12 11:14:15 2019][2937777.697825] .rt_throttled : 0 [Fri Apr 12 11:14:15 2019][2937777.702618] .rt_time : 0.000000 [Fri Apr 12 11:14:15 2019][2937777.708019] .rt_runtime : 950.000000 [Fri Apr 12 11:14:15 2019][2937777.713591] [Fri Apr 12 11:14:15 2019][2937777.715265] runnable tasks: [Fri Apr 12 11:14:15 2019][2937777.718237] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:15 2019][2937777.729192] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:15 2019][2937777.740064] watchdog/18 101 -11.979972 734430 0 0.000000 20061.924382 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.751405] migration/18 102 0.000000 17769 0 0.000000 169914.221945 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.762740] ksoftirqd/18 103 69694808.153239 28032444 120 0.000000 362817.482001 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.774076] kworker/18:0H 105 4011.204366 10 100 0.000000 0.067046 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.785447] ccp-3-q4 13527 3038.863635 2 120 0.000000 0.013535 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.796800] JS Sour~ Thread 18051 3963.610047 5 120 0.000000 0.081957 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.808148] rsyslogd 19283 69604831.325038 4912 120 0.000000 112.663445 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.819496] kworker/18:1H 19432 69694808.133969 24164177 100 0.000000 136873.326824 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.830839] kworker/u264:0 103155 57798632.255333 1100 100 0.000000 5.723703 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.842261] kworker/18:1 126303 69694720.309923 876522 120 0.000000 15872.806011 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.853700] ldlm_cb02_001 117263 69566629.834135 1381 120 0.000000 67.597259 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.865126] ll_mgs_0002 117284 69646137.269928 424307 120 0.000000 23090.455590 0.000000 2 / [Fri Apr 12 11:14:15 2019][2937777.876605] mdt02_009 117911 57838135.909347 2000212 120 0.000000 90437.478873 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937777.888042] mdt02_023 118218 57077538.659590 2060347 120 0.000000 77051.296306 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937777.899464] mdt02_026 118249 69694837.805697 17393703 120 0.000000 1075755.219953 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937777.910898] mdt02_035 118301 69693610.848473 24710022 120 0.000000 1591118.366513 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937777.922339] mdt02_043 118362 69608646.129873 11203743 120 0.000000 851629.055530 0.000000[2937777.932634] LustreError: 118537:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555092766, 90s ago); not entering recovery in server code, just going back to sleep ns: mdt-fir-MDT0000_UUID lock: ffff8b677dfe21c0/0x53d0f2240d8ea671 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2/0x0 rrc: 102 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 118537 timeout: 0 lvb_type: 0 [Fri Apr 12 11:14:16 2019][2937777.932638] LustreError: 118537:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) Skipped 8 previous similar messages [Fri Apr 12 11:14:16 2019] 2 / [Fri Apr 12 11:14:16 2019][2937777.984451] mdt02_048 118379 69652333.790594 23746743 120 0.000000 1553147.108434 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937777.995883] mdt02_053 118408 69693610.848021 26980553 120 0.000000 1542095.113387 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.007307] mdt02_059 118450 69694864.605989 24115514 120 0.000000 1414293.988270 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.018730] mdt02_077 118504 69694851.848839 12993061 120 0.000000 916099.293929 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.030157] mdt02_080 118510 69694826.620571 18071169 120 0.000000 1200986.202944 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.041589] mdt02_082 118514 56894272.409829 846612 120 0.000000 28989.057148 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.053012] mdt02_083 118516 69670192.871113 26843165 120 0.000000 1740372.412255 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.064436] mdt02_088 118526 69630199.933085 27656812 120 0.000000 1714395.959392 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.075858] mdt02_094 118539 69657536.252806 15239358 120 0.000000 1172252.357620 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.087281] mdt02_095 118541 69694647.283366 12074714 120 0.000000 902229.210363 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.098705] mdt02_102 118578 69581909.696258 10557716 120 0.000000 780566.183188 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.110125] mdt02_109 118588 69641160.738606 28900850 120 0.000000 1810124.248781 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.121556] mdt_io02_018 118645 57801562.513860 107 120 0.000000 6.515907 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.132983] mdt_io02_023 118663 57801361.383166 67 120 0.000000 4.364796 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.144412] mdt_io02_047 118752 57801361.088111 75 120 0.000000 4.947502 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.155837] mdt_io02_058 118797 57801361.721799 45 120 0.000000 2.196079 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.167276] mdt_io02_083 118900 57801361.747548 53 120 0.000000 2.948948 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.178708] mdt_io02_099 118952 57801361.257988 423054 120 0.000000 17299.731432 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.190137] mdt_io02_110 119000 57801360.374576 58 120 0.000000 2.088361 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.201564] mdt_rdpg02_006 119017 69688088.517420 25463078 120 0.000000 1165919.725472 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.212994] mdt_rdpg02_017 119069 69688088.517160 22386257 120 0.000000 1034155.898849 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.224417] mdt_rdpg02_018 119074 96826968.783667 14629295 120 0.000000 782350.481457 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.235852] mdt_rdpg02_035 119147 69688088.516217 24786598 120 0.000000 1117713.829400 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.247272] mdt_rdpg02_038 119160 69688088.516338 23854077 120 0.000000 1169952.145205 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.258705] mdt_rdpg02_057 119242 69659291.561428 15216212 120 0.000000 880755.167233 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.270184] kworker/18:2 46461 69615118.649959 2 120 0.000000 0.008546 0.000000 2 / [Fri Apr 12 11:14:16 2019][2937778.281534] [Fri Apr 12 11:14:16 2019][2937778.283206] cpu#19, 1996.203 MHz [Fri Apr 12 11:14:16 2019][2937778.286614] .nr_running : 0 [Fri Apr 12 11:14:16 2019][2937778.291407] .load : 0 [Fri Apr 12 11:14:16 2019][2937778.296200] .nr_switches : 9870315203 [Fri Apr 12 11:14:16 2019][2937778.301773] .nr_load_updates : 1806410082 [Fri Apr 12 11:14:16 2019][2937778.307354] .nr_uninterruptible : -31044 [Fri Apr 12 11:14:16 2019][2937778.312594] .next_balance : 7232.379060 [Fri Apr 12 11:14:16 2019][2937778.318254] .curr->pid : 0 [Fri Apr 12 11:14:16 2019][2937778.323048] .clock : 2937778322.830552 [Fri Apr 12 11:14:16 2019][2937778.329229] .cpu_load[0] : 1024 [Fri Apr 12 11:14:16 2019][2937778.334280] .cpu_load[1] : 729 [Fri Apr 12 11:14:16 2019][2937778.339247] .cpu_load[2] : 232 [Fri Apr 12 11:14:16 2019][2937778.344213] .cpu_load[3] : 253 [Fri Apr 12 11:14:16 2019][2937778.349178] .cpu_load[4] : 231 [Fri Apr 12 11:14:16 2019][2937778.354145] .avg_idle : 188792 [Fri Apr 12 11:14:16 2019][2937778.359372] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:16 2019][2937778.364697] [Fri Apr 12 11:14:16 2019][2937778.366371] cfs_rq[19]:/ [Fri Apr 12 11:14:16 2019][2937778.369083] .exec_clock : 0.000000 [Fri Apr 12 11:14:16 2019][2937778.374483] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:16 2019][2937778.379884] .min_vruntime : 75868800.214809 [Fri Apr 12 11:14:16 2019][2937778.385888] .max_vruntime : 0.000001 [Fri Apr 12 11:14:16 2019][2937778.391289] .spread : 0.000000 [Fri Apr 12 11:14:16 2019][2937778.396687] .spread0 : -40973997.993741 [Fri Apr 12 11:14:16 2019][2937778.402781] .nr_spread_over : 0 [Fri Apr 12 11:14:16 2019][2937778.407572] .nr_running : 0 [Fri Apr 12 11:14:16 2019][2937778.412365] .load : 0 [Fri Apr 12 11:14:16 2019][2937778.417159] .runnable_load_avg : 0 [Fri Apr 12 11:14:16 2019][2937778.421952] .blocked_load_avg : 126 [Fri Apr 12 11:14:16 2019][2937778.426918] .tg_load_avg : 0 [Fri Apr 12 11:14:16 2019][2937778.431710] .tg_load_contrib : 0 [Fri Apr 12 11:14:16 2019][2937778.436502] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:16 2019][2937778.441296] .tg->runnable_avg : 0 [Fri Apr 12 11:14:16 2019][2937778.446089] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:16 2019][2937778.450882] .throttled : 0 [Fri Apr 12 11:14:16 2019][2937778.455674] .throttle_count : 0 [Fri Apr 12 11:14:16 2019][2937778.460469] .avg->runnable_avg_sum : 6777 [Fri Apr 12 11:14:16 2019][2937778.465520] .avg->runnable_avg_period : 46120 [Fri Apr 12 11:14:16 2019][2937778.470669] [Fri Apr 12 11:14:16 2019][2937778.472340] rt_rq[19]:/ [Fri Apr 12 11:14:16 2019][2937778.474975] .rt_nr_running : 0 [Fri Apr 12 11:14:16 2019][2937778.479767] .rt_throttled : 0 [Fri Apr 12 11:14:16 2019][2937778.484561] .rt_time : 0.000000 [Fri Apr 12 11:14:16 2019][2937778.489961] .rt_runtime : 1000.000000 [Fri Apr 12 11:14:16 2019][2937778.495620] [Fri Apr 12 11:14:16 2019][2937778.497293] runnable tasks: [Fri Apr 12 11:14:16 2019][2937778.500264] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:16 2019][2937778.511211] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:16 2019][2937778.522084] watchdog/19 106 -11.980102 734430 0 0.000000 20799.517368 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.533424] migration/19 107 0.000000 10254 0 0.000000 111869.322291 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.544760] ksoftirqd/19 108 75868801.796050 18867683 120 0.000000 197047.822121 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.556096] kworker/19:0H 110 3509.119976 9 100 0.000000 0.029745 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.567470] kworker/u261:0 7394 1521.802725 4 100 0.000000 0.017823 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.578806] mlx5_fc 7399 1525.587829 2 100 0.000000 0.011061 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.590142] systemd-udevd 7899 57798266.806521 3539 120 0.000000 732.712957 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.601491] kworker/19:1H 17313 75868807.677097 17758635 100 0.000000 90111.110657 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.612832] auditd 17978 75867702.565341 399219 116 0.000000 14041.259470 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.624170] sssd 18059 75867195.344576 97931 120 0.000000 2826.421442 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.635507] dsm_sa_datamgrd 19150 3551.190316 2 120 0.000000 0.128484 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.646838] dsm_sa_datamgrd 19154 75868491.529280 586979 120 0.000000 21336.900147 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.658179] reader#1 19297 75868673.437427 356312 120 0.000000 37855.459550 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.669519] reader#2 19298 75868673.946204 319075 120 0.000000 117274.946317 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.680867] kworker/19:1 4033 75868772.316749 1470184 120 0.000000 31194.921851 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.692225] kiblnd_sd_03_00 117127 75868826.769565 1411184196 120 0.000000 11216122.262430 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.703736] kiblnd_sd_03_02 117129 60764190.992809 1411170071 120 0.000000 11212055.416440 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.715246] lnet_discovery 117132 75868746.187205 189583 120 0.000000 1513.849766 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.726693] ptlrpcd_03_01 117171 75868810.094021 2360661 120 0.000000 52122.682493 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.738127] ptlrpcd_03_03 117173 75868811.413652 2362242 120 0.000000 52565.742262 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.749558] mdt_rdpg03_000 117481 75868811.729537 13682588 120 0.000000 575814.820300 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.760997] osp-pre-10-2 117577 75868659.738773 40180 120 0.000000 972.550587 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.772424] osp-syn-43-2 117592 75868734.079863 235530 120 0.000000 6502.230411 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.783847] osp-pre-19-2 117595 75868799.688526 40505 120 0.000000 943.335529 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.795469] osp-pre-0-2 117603 75868744.810572 40220 120 0.000000 872.187119 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.806892] osp-syn-33-2 117620 75868538.559330 237086 120 0.000000 6454.197861 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.818312] osp-pre-33-2 117621 75868499.234755 41767 120 0.000000 986.543335 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.829753] osp-pre-34-2 117663 75868682.704705 40152 120 0.000000 930.250516 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.841189] osp-pre-3-0 117719 75868569.755520 36804 120 0.000000 617.078690 0.000000 3 / [Fri Apr 12 11:14:16 2019][2937778.852614] osp-pre-0-0 117755 75868527.025239 38836 120 0.000000 792.143172 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.864039] osp-pre-44-0 117759 75868645.758676 39894 120 0.000000 833.403419 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.875464] osp-pre-41-0 117767 75868612.257146 40152 120 0.000000 867.860361 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.886887] osp-syn-23-0 117770 75855866.378905 75615 120 0.000000 2340.936978 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.898310] osp-pre-45-0 117781 75868756.590122 40641 120 0.000000 863.530321 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.909731] osp-pre-6-0 117783 75868627.885236 39296 120 0.000000 842.426081 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.921186] osp-syn-35-0 117808 75275525.819869 74981 120 0.000000 2306.785483 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.932611] osp-syn-28-0 117810 75855750.776833 74748 120 0.000000 2347.391369 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.944056] mdt03_018 118246 72225024.880149 12343395 120 0.000000 738148.576214 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.955499] mdt_rdpg03_015 118521 75854313.809361 16046289 120 0.000000 646989.599736 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.966924] mdt_rdpg03_017 118525 75868711.392328 16678010 120 0.000000 714725.269053 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.978349] mdt03_058 118562 75868711.392168 6747053 120 0.000000 298571.202910 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937778.989771] mdt_io03_005 118602 75861228.862262 2414510 120 0.000000 177147.705248 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.001199] mdt_io03_012 118646 57798627.173793 264272 120 0.000000 9751.426968 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.012627] mdt03_078 118667 55184985.872990 3317909 120 0.000000 118631.768176 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.024058] mdt_io03_019 118679 57798512.652427 45 120 0.000000 129.247562 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.035496] mdt_io03_043 118779 75861495.461449 2231799 120 0.000000 150095.593424 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.046921] mdt_io03_046 118798 75866986.243553 667580 120 0.000000 68579.746130 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.058349] mdt_io03_057 118838 57798510.857186 56 120 0.000000 1.812721 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.069775] mdt_io03_060 118843 74994048.726615 2207885 120 0.000000 146245.865885 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.081207] mdt_io03_062 118846 75868839.284743 2143947 120 0.000000 158647.446290 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.092635] mdt_io03_072 118884 57798510.832687 26 120 0.000000 1.207792 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.104070] mdt_io03_076 118897 57798510.967795 49 120 0.000000 1.833295 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.115651] mdt_io03_082 118917 57798510.730484 29 120 0.000000 1.046255 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.127075] mdt_io03_084 118932 57798510.902214 54 120 0.000000 2.366888 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.138503] mdt_io03_085 118938 57798510.823892 28 120 0.000000 1.145405 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.149956] mdt_rdpg03_030 119124 75864856.060003 15759254 120 0.000000 620132.556966 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.161409] mdt_rdpg03_044 119244 75864856.059652 16943200 120 0.000000 603581.765513 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.172841] mdt_rdpg03_049 119257 75854313.809161 14625966 120 0.000000 688842.814339 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.184272] ldlm_cn03_012 123641 61764730.084006 46403394 120 0.000000 903520.458279 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.195707] osp-syn-39-3 2236 75868115.157988 9578 120 0.000000 316.214276 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.207040] osp-syn-14-3 2244 75868251.238278 9565 120 0.000000 286.145472 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.218375] osp-pre-42-3 2251 75868672.303548 28036 120 0.000000 588.752320 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.229712] osp-syn-19-3 2258 75868645.814662 9552 120 0.000000 289.031581 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.241048] osp-syn-18-3 2260 75867384.294019 9439 120 0.000000 285.201003 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.252384] osp-pre-18-3 2261 75868496.446064 27874 120 0.000000 503.606134 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.263722] osp-syn-0-3 2266 75868810.135631 9568 120 0.000000 316.159087 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.275058] osp-syn-44-3 2270 75868672.664712 9581 120 0.000000 301.592129 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.286392] osp-syn-21-3 2274 75866514.689175 9424 120 0.000000 280.578661 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.297728] osp-syn-16-3 2276 75867180.942795 9550 120 0.000000 297.836298 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.309065] osp-syn-33-3 2284 75868672.501305 9539 120 0.000000 296.248165 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.320400] osp-syn-20-3 2286 75868672.413708 9490 120 0.000000 294.886509 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.331739] osp-pre-45-3 2293 75868852.692693 27855 120 0.000000 653.318077 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.343074] osp-syn-24-3 2296 75867979.853128 9379 120 0.000000 274.468306 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.354411] osp-syn-47-3 2298 75866514.681961 9465 120 0.000000 310.795468 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.365749] osp-pre-38-3 2301 75868492.423228 28020 120 0.000000 579.030218 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.377083] osp-syn-29-3 2304 75868672.503849 9556 120 0.000000 306.251127 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.388419] osp-pre-1-3 2307 75868715.615236 28005 120 0.000000 538.915412 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.399756] osp-syn-30-3 2324 75868527.089961 9465 120 0.000000 302.386798 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.411091] osp-syn-34-3 2326 75868672.506134 9379 120 0.000000 276.429777 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.422427] lquota_wb_fir-M 2345 75866066.683758 2327 120 0.000000 28.087169 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.433767] osp-pre-12-1 2391 75868516.305434 27716 120 0.000000 633.018700 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.445101] osp-pre-10-1 2399 75868826.957880 27716 120 0.000000 543.609575 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.456439] osp-syn-25-1 2420 75865323.302545 20012 120 0.000000 514.984638 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.467772] osp-pre-17-1 2423 75868781.670646 27518 120 0.000000 580.358780 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.479109] osp-syn-41-1 2436 75865538.512947 19826 120 0.000000 516.383138 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.490446] osp-pre-29-1 2463 75868626.471394 27842 120 0.000000 613.693226 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.501783] osp-pre-1-1 2465 75868514.936855 28051 120 0.000000 671.710844 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.513119] osp-syn-34-1 2484 75865896.846702 19782 120 0.000000 500.719806 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.524457] mdt_io03_095 3066 71835179.729382 2488514 120 0.000000 171053.660921 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.535804] kworker/19:2 43802 72161820.820131 3 120 0.000000 0.016842 0.000000 3 / [Fri Apr 12 11:14:17 2019][2937779.547156] [Fri Apr 12 11:14:17 2019][2937779.548827] cpu#20, 1996.203 MHz [Fri Apr 12 11:14:17 2019][2937779.552235] .nr_running : 0 [Fri Apr 12 11:14:17 2019][2937779.557030] .load : 1024 [Fri Apr 12 11:14:17 2019][2937779.562090] .nr_switches : 15287371707 [Fri Apr 12 11:14:17 2019][2937779.567751] .nr_load_updates : 2127635285 [Fri Apr 12 11:14:17 2019][2937779.573330] .nr_uninterruptible : 758914 [Fri Apr 12 11:14:17 2019][2937779.578556] .next_balance : 7232.380322 [Fri Apr 12 11:14:17 2019][2937779.584237] .curr->pid : 0 [Fri Apr 12 11:14:17 2019][2937779.589036] .clock : 2937779588.931554 [Fri Apr 12 11:14:17 2019][2937779.595224] .cpu_load[0] : 1024 [Fri Apr 12 11:14:17 2019][2937779.600284] .cpu_load[1] : 547 [Fri Apr 12 11:14:17 2019][2937779.605251] .cpu_load[2] : 518 [Fri Apr 12 11:14:17 2019][2937779.610218] .cpu_load[3] : 591 [Fri Apr 12 11:14:17 2019][2937779.615183] .cpu_load[4] : 1564 [Fri Apr 12 11:14:17 2019][2937779.620236] .avg_idle : 44815 [Fri Apr 12 11:14:17 2019][2937779.625374] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:17 2019][2937779.630687] [Fri Apr 12 11:14:17 2019][2937779.632357] cfs_rq[20]:/ [Fri Apr 12 11:14:17 2019][2937779.635070] .exec_clock : 0.000000 [Fri Apr 12 11:14:17 2019][2937779.640469] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:17 2019][2937779.645869] .min_vruntime : 109944136.770607 [Fri Apr 12 11:14:17 2019][2937779.651963] .max_vruntime : 0.000001 [Fri Apr 12 11:14:17 2019][2937779.657362] .spread : 0.000000 [Fri Apr 12 11:14:17 2019][2937779.662761] .spread0 : -6898907.065954 [Fri Apr 12 11:14:17 2019][2937779.668766] .nr_spread_over : 0 [Fri Apr 12 11:14:17 2019][2937779.673560] .nr_running : 0 [Fri Apr 12 11:14:17 2019][2937779.678353] .load : 2048 [Fri Apr 12 11:14:17 2019][2937779.683405] .runnable_load_avg : 0 [Fri Apr 12 11:14:17 2019][2937779.688199] .blocked_load_avg : 3182 [Fri Apr 12 11:14:17 2019][2937779.693250] .tg_load_avg : 0 [Fri Apr 12 11:14:17 2019][2937779.698044] .tg_load_contrib : 0 [Fri Apr 12 11:14:17 2019][2937779.702837] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:17 2019][2937779.707630] .tg->runnable_avg : 0 [Fri Apr 12 11:14:17 2019][2937779.712422] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:17 2019][2937779.717215] .throttled : 0 [Fri Apr 12 11:14:17 2019][2937779.722007] .throttle_count : 0 [Fri Apr 12 11:14:17 2019][2937779.726801] .avg->runnable_avg_sum : 25179 [Fri Apr 12 11:14:17 2019][2937779.731940] .avg->runnable_avg_period : 46171 [Fri Apr 12 11:14:17 2019][2937779.737086] [Fri Apr 12 11:14:17 2019][2937779.738760] rt_rq[20]:/ [Fri Apr 12 11:14:17 2019][2937779.741385] .rt_nr_running : 0 [Fri Apr 12 11:14:17 2019][2937779.746179] .rt_throttled : 0 [Fri Apr 12 11:14:17 2019][2937779.750971] .rt_time : 0.000000 [Fri Apr 12 11:14:17 2019][2937779.756372] .rt_runtime : 950.000000 [Fri Apr 12 11:14:17 2019][2937779.761944] [Fri Apr 12 11:14:17 2019][2937779.763617] runnable tasks: [Fri Apr 12 11:14:17 2019][2937779.766590] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:17 2019][2937779.777536] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:17 2019][2937779.788410] watchdog/20 111 -11.982647 734431 0 0.000000 39681.747434 0.000000 0 / [Fri Apr 12 11:14:17 2019][2937779.799749] migration/20 112 0.000000 31579 0 0.000000 414892.783648 0.000000 0 / [Fri Apr 12 11:14:17 2019][2937779.811085] ksoftirqd/20 113 109944163.315327 30300698 120 0.000000 415658.828762 0.000000 0 / [Fri Apr 12 11:14:17 2019][2937779.822421] kworker/20:0H 115 2853.027804 9 100 0.000000 0.027643 0.000000 0 / [Fri Apr 12 11:14:17 2019][2937779.833798] kworker/20:1H 16842 109944166.915262 28354431 100 0.000000 180441.515009 0.000000 0 / [Fri Apr 12 11:14:17 2019][2937779.845151] dsm_sa_datamgrd 19168 3578.145966 34 120 0.000000 2.907912 0.000000 0 / [Fri Apr 12 11:14:17 2019][2937779.856499] ptlrpc_hr00_001 117074 109944116.054760 2396467 120 0.000000 69733.448826 0.000000 0 / [Fri Apr 12 11:14:17 2019][2937779.867925] kiblnd_sd_00_01 117116 109944184.139894 2268918283 120 0.000000 19918069.774941 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.879435] ptlrpcd_00_07 117141 109944155.750138 879690 120 0.000000 22445.711173 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.890894] osp-syn-45-2 117628 109943171.444677 235730 120 0.000000 6250.812723 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.902319] osp-syn-30-2 117660 109943478.996821 236544 120 0.000000 6649.221434 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.913753] osp-pre-9-0 117741 109944172.368621 39318 120 0.000000 1047.374388 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.925208] osp-syn-41-0 117766 109905056.664455 74985 120 0.000000 2181.007351 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.936665] mdt00_019 118183 109899606.902132 16608278 120 0.000000 1183796.663785 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.948095] mdt00_023 118206 93204965.751893 3040235 120 0.000000 105972.429243 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.959515] mdt00_024 118208 92820970.033967 1066641 120 0.000000 30255.940274 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.970941] mdt00_032 118226 109882214.615131 14679074 120 0.000000 1082820.738340 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.982361] mdt00_045 118267 109880814.134531 34565379 120 0.000000 1837246.859288 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937779.993781] mdt00_046 118270 109932481.058242 20911168 120 0.000000 1341607.358363 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.005208] mdt00_052 118287 94097653.598648 236714 120 0.000000 8832.984511 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.016631] mdt00_059 118311 109935065.370307 26635620 120 0.000000 1400057.597952 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.028050] mdt00_060 118314 109944202.188088 17589497 120 0.000000 1113235.652219 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.039483] mdt_io00_004 118416 109941802.649233 46546 120 0.000000 3776.621553 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.050908] mdt00_079 118439 109944172.353843 23053406 120 0.000000 1476080.689012 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.062332] mdt00_089 118474 94097652.695903 5790757 120 0.000000 209387.390240 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.073758] mdt00_097 118549 109816093.685914 17878736 120 0.000000 1225279.476091 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.085183] mdt00_101 118555 109944175.979520 27234562 120 0.000000 1594850.162112 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.096611] mdt_io00_010 118608 95016636.366886 73 120 0.000000 3.965980 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.108031] mdt_io00_013 118618 95016450.357650 4705 120 0.000000 115.318741 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.119455] mdt_io00_022 118650 109937816.791988 2618032 120 0.000000 177350.318597 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.130883] mdt_io00_035 118734 95016450.351288 44 120 0.000000 3.318875 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.142310] mdt_io00_039 118768 95016452.582408 71 120 0.000000 4.164306 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.153745] mdt_io00_062 118876 95016451.577654 53 120 0.000000 2.957846 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.165174] mdt_io00_067 118899 95016636.288016 210 120 0.000000 8.152181 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.176596] mdt_io00_070 118916 95016636.844285 212832 120 0.000000 10111.181857 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.188019] mdt_io00_074 118937 95016638.591015 117 120 0.000000 3.816753 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.199440] mdt_io00_076 118947 95016636.387506 177878 120 0.000000 9366.608802 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.210864] mdt_io00_081 118962 109901196.702564 1185632 120 0.000000 96567.707580 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.222289] mdt_io00_088 118991 95016453.158812 30 120 0.000000 0.892468 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.233710] mdt_io00_095 119018 95016451.329983 46 120 0.000000 2.134524 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.245156] mdt_rdpg00_034 119156 109933126.629657 25899954 120 0.000000 1065734.399297 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.256637] kworker/20:0 20142 109944231.564949 121808 120 0.000000 9391.432784 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.267981] kworker/20:2 42444 108771943.569939 2 120 0.000000 0.008506 0.000000 0 / [Fri Apr 12 11:14:18 2019][2937780.279354] [Fri Apr 12 11:14:18 2019][2937780.281030] cpu#21, 1996.203 MHz [Fri Apr 12 11:14:18 2019][2937780.284439] .nr_running : 3 [Fri Apr 12 11:14:18 2019][2937780.289232] .load : 0 [Fri Apr 12 11:14:18 2019][2937780.294025] .nr_switches : 18314792533 [Fri Apr 12 11:14:18 2019][2937780.299682] .nr_load_updates : 2171780063 [Fri Apr 12 11:14:18 2019][2937780.305257] .nr_uninterruptible : -88497 [Fri Apr 12 11:14:18 2019][2937780.310491] .next_balance : 7232.381054 [Fri Apr 12 11:14:18 2019][2937780.316151] .curr->pid : 118437 [Fri Apr 12 11:14:18 2019][2937780.321376] .clock : 2937780321.274587 [Fri Apr 12 11:14:18 2019][2937780.327555] .cpu_load[0] : 0 [Fri Apr 12 11:14:18 2019][2937780.332347] .cpu_load[1] : 265 [Fri Apr 12 11:14:18 2019][2937780.337316] .cpu_load[2] : 107 [Fri Apr 12 11:14:18 2019][2937780.342282] .cpu_load[3] : 390 [Fri Apr 12 11:14:18 2019][2937780.347248] .cpu_load[4] : 468 [Fri Apr 12 11:14:18 2019][2937780.352214] .avg_idle : 118854 [Fri Apr 12 11:14:18 2019][2937780.357438] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:18 2019][2937780.362751] [Fri Apr 12 11:14:18 2019][2937780.364423] cfs_rq[21]:/ [Fri Apr 12 11:14:18 2019][2937780.367134] .exec_clock : 0.000000 [Fri Apr 12 11:14:18 2019][2937780.372533] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:18 2019][2937780.377932] .min_vruntime : 129408581.254244 [Fri Apr 12 11:14:18 2019][2937780.384025] .max_vruntime : 0.000001 [Fri Apr 12 11:14:18 2019][2937780.389425] .spread : 0.000000 [Fri Apr 12 11:14:18 2019][2937780.394826] .spread0 : 12565417.207919 [Fri Apr 12 11:14:18 2019][2937780.400830] .nr_spread_over : 0 [Fri Apr 12 11:14:18 2019][2937780.405624] .nr_running : 0 [Fri Apr 12 11:14:18 2019][2937780.410416] .load : 1024 [Fri Apr 12 11:14:18 2019][2937780.415471] .runnable_load_avg : 253 [Fri Apr 12 11:14:18 2019][2937780.420435] .blocked_load_avg : 530 [Fri Apr 12 11:14:18 2019][2937780.425400] .tg_load_avg : 0 [Fri Apr 12 11:14:18 2019][2937780.430195] .tg_load_contrib : 0 [Fri Apr 12 11:14:18 2019][2937780.434986] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:18 2019][2937780.439780] .tg->runnable_avg : 0 [Fri Apr 12 11:14:18 2019][2937780.444571] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:18 2019][2937780.449365] .throttled : 0 [Fri Apr 12 11:14:18 2019][2937780.454159] .throttle_count : 0 [Fri Apr 12 11:14:18 2019][2937780.458951] .avg->runnable_avg_sum : 20308 [Fri Apr 12 11:14:18 2019][2937780.464091] .avg->runnable_avg_period : 46624 [Fri Apr 12 11:14:18 2019][2937780.469237] [Fri Apr 12 11:14:18 2019][2937780.470912] rt_rq[21]:/ [Fri Apr 12 11:14:18 2019][2937780.473537] .rt_nr_running : 0 [Fri Apr 12 11:14:18 2019][2937780.478330] .rt_throttled : 0 [Fri Apr 12 11:14:18 2019][2937780.483123] .rt_time : 0.000000 [Fri Apr 12 11:14:18 2019][2937780.488522] .rt_runtime : 1000.000000 [Fri Apr 12 11:14:18 2019][2937780.494182] [Fri Apr 12 11:14:18 2019][2937780.495855] runnable tasks: [Fri Apr 12 11:14:18 2019][2937780.498827] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:18 2019][2937780.509772] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:18 2019][2937780.520646] watchdog/21 116 -11.980433 734431 0 0.000000 61448.490055 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.531986] migration/21 117 0.000000 48795 0 0.000000 631217.690953 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.543321] ksoftirqd/21 118 129408594.465540 43659572 120 0.000000 591134.427239 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.554659] kworker/21:0H 120 3461.767340 9 100 0.000000 0.039565 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.566035] kworker/21:1H 16840 129408598.443476 39085613 100 0.000000 234304.534052 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.577377] kdmflush 17183 3546.348218 2 100 0.000000 0.013004 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.588711] bioset 17185 3546.351875 2 100 0.000000 0.012303 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.600047] kdmflush 17258 3546.594477 2 100 0.000000 0.011703 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.611383] bioset 17261 3546.597547 2 100 0.000000 0.011823 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.622722] kdmflush 17539 3577.378915 2 100 0.000000 0.013055 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.634077] kworker/21:0 27581 129408555.793703 1121122 120 0.000000 21524.336416 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.645421] ptlrpcd_01_01 117147 129406416.603190 676331 120 0.000000 15841.761996 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.656840] ptlrpcd_01_09 117155 129406349.131475 673982 120 0.000000 16028.058115 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.668330] mdt01_020 118182 129408592.396447 27049011 120 0.000000 1699273.505570 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.679758] mdt01_022 118187 129408614.124774 29524563 120 0.000000 1932502.875701 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.691185] mdt01_031 118221 125774645.474207 12887203 120 0.000000 862661.734340 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.702612] mdt01_034 118237 129367319.490092 31163083 120 0.000000 1953915.827626 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.714036] mdt01_040 118256 114886004.304500 358316 120 0.000000 13843.872790 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.725458] mdt01_044 118269 129408565.737475 25187609 120 0.000000 1754521.134560 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.736883] mdt01_056 118302 114886004.297756 611199 120 0.000000 23393.952260 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.748302] mdt01_059 118310 129386585.521924 17653078 120 0.000000 1355278.468492 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.759726] R mdt01_062 118317 129408631.712520 34761311 120 0.000000 2202513.932696 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.771149] mdt01_070 118332 129317966.692400 24770612 120 0.000000 1750004.541686 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.782575] mdt01_083 118350 129408558.766090 27561719 120 0.000000 1748124.596502 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.793996] mdt01_090 118366 129325928.719610 23267153 120 0.000000 1620117.745059 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.805422] mdt01_108 118428 123809992.290776 11118053 120 0.000000 693270.437753 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.816852] mdt_io01_009 118606 129408558.008271 1567570 120 0.000000 135332.170504 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.828274] mdt_io01_012 118637 115526260.212065 1824 120 0.000000 38.650734 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.839709] mdt_io01_029 118783 120575900.985689 390899 120 0.000000 17787.954009 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.851145] mdt_io01_043 118858 120575900.985038 26969 120 0.000000 1117.161358 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.862584] mdt_io01_058 118965 129408579.546859 2319028 120 0.000000 181880.431171 0.000000 1 / [Fri Apr 12 11:14:18 2019][2937780.874013] mdt_rdpg01_006 119022 129408666.207754 27508420 120 0.000000 1333583.922641 0.000000 1 / [Fri Apr 12 11:14:19 2019][2937780.885438] mdt_io01_072 119029 129380910.840773 1638171 120 0.000000 154754.458283 0.000000 1 / [Fri Apr 12 11:14:19 2019][2937780.896879] mdt_io01_095 119138 129408558.742476 1633837 120 0.000000 85936.552816 0.000000 1 / [Fri Apr 12 11:14:19 2019][2937780.908324] ldlm_cn01_011 122529 129408555.052844 52248391 120 0.000000 1313022.949837 0.000000 1 / [Fri Apr 12 11:14:19 2019][2937780.919754] ldlm_cn01_021 126372 129408555.053224 45127393 120 0.000000 1169029.596273 0.000000 1 / [Fri Apr 12 11:14:19 2019][2937780.931185] ldlm_bl_13 128946 129408626.059241 700743 120 0.000000 15536.371023 0.000000 1 / [Fri Apr 12 11:14:19 2019][2937780.942642] kworker/21:1 42367 127768339.355302 2 120 0.000000 0.006722 0.000000 1 / [Fri Apr 12 11:14:19 2019][2937780.953995] [Fri Apr 12 11:14:19 2019][2937780.955669] cpu#22, 1996.203 MHz [Fri Apr 12 11:14:19 2019][2937780.959075] .nr_running : 1 [Fri Apr 12 11:14:19 2019][2937780.963869] .load : 2048 [Fri Apr 12 11:14:19 2019][2937780.968922] .nr_switches : 17250175839 [Fri Apr 12 11:14:19 2019][2937780.974582] .nr_load_updates : 1999146632 [Fri Apr 12 11:14:19 2019][2937780.980154] .nr_uninterruptible : -140707 [Fri Apr 12 11:14:19 2019][2937780.985467] .next_balance : 7232.381732 [Fri Apr 12 11:14:19 2019][2937780.991125] .curr->pid : 0 [Fri Apr 12 11:14:19 2019][2937780.995919] .clock : 2937780995.765147 [Fri Apr 12 11:14:19 2019][2937781.002098] .cpu_load[0] : 0 [Fri Apr 12 11:14:19 2019][2937781.006890] .cpu_load[1] : 269 [Fri Apr 12 11:14:19 2019][2937781.011856] .cpu_load[2] : 410 [Fri Apr 12 11:14:19 2019][2937781.016825] .cpu_load[3] : 730 [Fri Apr 12 11:14:19 2019][2937781.021809] .cpu_load[4] : 644 [Fri Apr 12 11:14:19 2019][2937781.026783] .avg_idle : 323304 [Fri Apr 12 11:14:19 2019][2937781.032021] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:19 2019][2937781.037339] [Fri Apr 12 11:14:19 2019][2937781.039017] cfs_rq[22]:/ [Fri Apr 12 11:14:19 2019][2937781.041728] .exec_clock : 0.000000 [Fri Apr 12 11:14:19 2019][2937781.047139] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:19 2019][2937781.052547] .min_vruntime : 87966803.919593 [Fri Apr 12 11:14:19 2019][2937781.058569] .max_vruntime : 0.000001 [Fri Apr 12 11:14:19 2019][2937781.063969] .spread : 0.000000 [Fri Apr 12 11:14:19 2019][2937781.069378] .spread0 : -28876436.063971 [Fri Apr 12 11:14:19 2019][2937781.075477] .nr_spread_over : 0 [Fri Apr 12 11:14:19 2019][2937781.080270] .nr_running : 1 [Fri Apr 12 11:14:19 2019][2937781.085063] .load : 0 [Fri Apr 12 11:14:19 2019][2937781.089857] .runnable_load_avg : 115 [Fri Apr 12 11:14:19 2019][2937781.094822] .blocked_load_avg : 881 [Fri Apr 12 11:14:19 2019][2937781.099789] .tg_load_avg : 0 [Fri Apr 12 11:14:19 2019][2937781.104580] .tg_load_contrib : 0 [Fri Apr 12 11:14:19 2019][2937781.109373] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:19 2019][2937781.114166] .tg->runnable_avg : 0 [Fri Apr 12 11:14:19 2019][2937781.118959] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:19 2019][2937781.123752] .throttled : 0 [Fri Apr 12 11:14:19 2019][2937781.128545] .throttle_count : 0 [Fri Apr 12 11:14:19 2019][2937781.133337] .avg->runnable_avg_sum : 14965 [Fri Apr 12 11:14:19 2019][2937781.138476] .avg->runnable_avg_period : 46199 [Fri Apr 12 11:14:19 2019][2937781.143623] [Fri Apr 12 11:14:19 2019][2937781.145298] rt_rq[22]:/ [Fri Apr 12 11:14:19 2019][2937781.147924] .rt_nr_running : 0 [Fri Apr 12 11:14:19 2019][2937781.152716] .rt_throttled : 0 [Fri Apr 12 11:14:19 2019][2937781.157509] .rt_time : 0.000000 [Fri Apr 12 11:14:19 2019][2937781.162909] .rt_runtime : 950.000000 [Fri Apr 12 11:14:19 2019][2937781.168508] [Fri Apr 12 11:14:19 2019][2937781.170195] runnable tasks: [Fri Apr 12 11:14:19 2019][2937781.173170] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:19 2019][2937781.184117] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:19 2019][2937781.194990] watchdog/22 121 -11.980072 734431 0 0.000000 31950.168690 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.206328] migration/22 122 0.000000 32988 0 0.000000 487516.488985 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.217664] ksoftirqd/22 123 87966818.458790 31497452 120 0.000000 650367.910148 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.229010] kworker/22:0H 125 3687.703707 9 100 0.000000 0.034675 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.240376] ccp-3-q2 13519 2943.965975 2 120 0.000000 0.008566 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.251859] ccp-4-q3 13542 2935.896906 2 120 0.000000 0.012544 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.263206] kmpath_handlerd 14019 3055.252907 2 100 0.000000 0.013495 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.274551] kworker/22:1H 16889 87966819.026086 21104767 100 0.000000 129557.017736 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.285908] chronyd 18083 87964574.119396 15886 120 0.000000 1163.190644 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.297259] writer#0 19291 87966242.633279 15220325 120 0.000000 235905.153908 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.308648] kworker/22:0 38528 86982339.103684 37659 120 0.000000 320.478279 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.320015] ptlrpc_hr02_005 117091 87966806.393502 2387983 120 0.000000 75482.394585 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.331461] ptlrpcd_02_09 117167 87966809.227617 856907 120 0.000000 21597.563084 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.342888] ldlm_cb02_000 117261 79774133.928776 295 120 0.000000 6.540771 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.354311] jbd2/dm-0-8 117446 87966697.138143 322914 120 0.000000 770991.286756 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.365741] mdt_io02_000 117513 87966786.781629 2990322 120 0.000000 241260.575797 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.377195] osp-pre-31-0 117737 87966309.466408 39347 120 0.000000 872.948012 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.388636] ldlm_cn02_006 117901 87966818.861948 46114725 120 0.000000 1111224.802292 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.400077] ll_mgs_0007 117905 87966698.440931 2981836 120 0.000000 101540.237587 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.411512] mdt02_018 118194 87966827.699772 14055241 120 0.000000 970323.013529 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.422949] mdt02_019 118198 87889666.324890 18227003 120 0.000000 1262033.120681 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.434372] mdt02_020 118204 77588622.706025 1546555 120 0.000000 58917.834681 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.445800] mdt02_024 118238 78369433.887197 492 120 0.000000 36.830979 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.457239] mdt02_046 118367 87966831.242661 19820698 120 0.000000 1383471.008100 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.468660] mdt02_052 118405 87965584.769753 20257662 120 0.000000 1414724.426359 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.480091] mdt02_084 118517 87955181.824473 22677673 120 0.000000 1365288.216852 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.491518] mdt_io02_008 118601 87966831.245126 3777248 120 0.000000 297591.040449 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.502946] mdt_io02_014 118628 87463273.401614 2347590 120 0.000000 161548.254708 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.514368] mdt_io02_016 118638 87965850.995950 2106854 120 0.000000 150743.314672 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.525794] mdt_io02_020 118652 83401476.805682 1737075 120 0.000000 115666.373702 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.537222] mdt_io02_021 118655 87966789.144219 1788035 120 0.000000 126917.000108 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.548646] mdt_io02_022 118657 87463273.401825 1709458 120 0.000000 106428.553744 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.560081] mdt_io02_031 118692 87457568.669167 482682 120 0.000000 21556.570391 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.571500] mdt_io02_033 118700 87916587.359295 1960972 120 0.000000 175506.401860 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.582923] mdt_io02_034 118701 87942983.340286 2950158 120 0.000000 233616.314436 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.594346] mdt_io02_035 118706 78364936.274769 45026 120 0.000000 1835.948997 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.605767] mdt_io02_037 118711 87962914.601193 1808545 120 0.000000 165846.388438 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.617190] mdt_io02_039 118715 87942983.340096 770895 120 0.000000 46963.435365 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.628622] mdt_io02_055 118789 87916587.352031 8409 120 0.000000 658.245474 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.640056] mdt_io02_065 118817 78364839.317656 103 120 0.000000 3.872652 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.651602] mdt_io02_078 118885 78364839.135393 62 120 0.000000 1.861204 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.663021] mdt_io02_079 118886 87965851.100797 2896310 120 0.000000 214310.974618 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.674449] mdt_io02_088 118912 78364935.327163 68 120 0.000000 3.697189 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.685867] mdt_io02_089 118915 78364838.379248 60 120 0.000000 2.531323 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.697290] mdt_io02_090 118918 78364840.009909 45 120 0.000000 3.649489 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.708723] mdt_io02_109 118999 87942983.339904 3252154 120 0.000000 259496.897386 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.720153] mdt_rdpg02_011 119046 87966855.151516 19413612 120 0.000000 1002437.742089 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.731582] mdt_rdpg02_016 119067 87966866.477844 21559149 120 0.000000 979976.008330 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.743010] mdt_rdpg02_026 119096 87955699.811625 21651455 120 0.000000 1092353.474470 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.754449] mdt_rdpg02_036 119155 87939209.135270 22268802 120 0.000000 1191291.298255 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.765882] mdt_rdpg02_041 119167 87966870.401884 17930051 120 0.000000 943086.212482 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.777305] mdt_rdpg02_046 119189 87961511.468246 22721762 120 0.000000 1047079.438472 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.788770] ldlm_cn02_011 123638 87966818.862529 40710696 120 0.000000 1008139.787218 0.000000 2 / [Fri Apr 12 11:14:19 2019][2937781.800199] ldlm_cn02_017 129939 87966862.824294 35773290 120 0.000000 925725.805619 0.000000 2 / [Fri Apr 12 11:14:20 2019][2937781.811623] ldlm_cn02_020 129942 87966818.861547 37682384 120 0.000000 942591.159096 0.000000 2 / [Fri Apr 12 11:14:20 2019][2937781.823069] ldlm_cn02_022 6477 87966865.209610 33034923 120 0.000000 838464.838103 0.000000 2 / [Fri Apr 12 11:14:20 2019][2937781.834408] ldlm_cn02_025 6480 87966818.862219 32439819 120 0.000000 841626.428263 0.000000 2 / [Fri Apr 12 11:14:20 2019][2937781.845744] ldlm_cn02_026 6481 87966818.865224 30050481 120 0.000000 791188.518995 0.000000 2 / [Fri Apr 12 11:14:20 2019][2937781.857083] ldlm_cn02_029 6495 87966818.861708 28298239 120 0.000000 751901.065827 0.000000 2 / [Fri Apr 12 11:14:20 2019][2937781.868436] kworker/22:1 43552 87966818.465373 14947 120 0.000000 2701.126568 0.000000 2 / [Fri Apr 12 11:14:20 2019][2937781.879801] kworker/u259:0 45187 87964380.378269 4950 120 0.000000 1182.027691 0.000000 2 / [Fri Apr 12 11:14:20 2019][2937781.891392] [Fri Apr 12 11:14:20 2019][2937781.893283] cpu#23, 1996.203 MHz [Fri Apr 12 11:14:20 2019][2937781.896876] .nr_running : 0 [Fri Apr 12 11:14:20 2019][2937781.901675] .load : 0 [Fri Apr 12 11:14:20 2019][2937781.906468] .nr_switches : 11594041964 [Fri Apr 12 11:14:20 2019][2937781.912137] .nr_load_updates : 1823341660 [Fri Apr 12 11:14:20 2019][2937781.917753] .nr_uninterruptible : -28012 [Fri Apr 12 11:14:20 2019][2937781.923079] .next_balance : 7232.382668 [Fri Apr 12 11:14:20 2019][2937781.928840] .curr->pid : 0 [Fri Apr 12 11:14:20 2019][2937781.933661] .clock : 2937781933.566022 [Fri Apr 12 11:14:20 2019][2937781.939844] .cpu_load[0] : 0 [Fri Apr 12 11:14:20 2019][2937781.944640] .cpu_load[1] : 1 [Fri Apr 12 11:14:20 2019][2937781.949440] .cpu_load[2] : 4 [Fri Apr 12 11:14:20 2019][2937781.954242] .cpu_load[3] : 257 [Fri Apr 12 11:14:20 2019][2937781.959216] .cpu_load[4] : 191 [Fri Apr 12 11:14:20 2019][2937781.964191] .avg_idle : 498008 [Fri Apr 12 11:14:20 2019][2937781.969425] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:20 2019][2937781.974746] [Fri Apr 12 11:14:20 2019][2937781.976416] cfs_rq[23]:/ [Fri Apr 12 11:14:20 2019][2937781.979127] .exec_clock : 0.000000 [Fri Apr 12 11:14:20 2019][2937781.984529] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:20 2019][2937781.989928] .min_vruntime : 83624594.464347 [Fri Apr 12 11:14:20 2019][2937781.995934] .max_vruntime : 0.000001 [Fri Apr 12 11:14:20 2019][2937782.001334] .spread : 0.000000 [Fri Apr 12 11:14:20 2019][2937782.006733] .spread0 : -33218757.490426 [Fri Apr 12 11:14:20 2019][2937782.012826] .nr_spread_over : 0 [Fri Apr 12 11:14:20 2019][2937782.017618] .nr_running : 0 [Fri Apr 12 11:14:20 2019][2937782.022410] .load : 0 [Fri Apr 12 11:14:20 2019][2937782.027204] .runnable_load_avg : 0 [Fri Apr 12 11:14:20 2019][2937782.031996] .blocked_load_avg : 166 [Fri Apr 12 11:14:20 2019][2937782.036970] .tg_load_avg : 0 [Fri Apr 12 11:14:20 2019][2937782.041781] .tg_load_contrib : 0 [Fri Apr 12 11:14:20 2019][2937782.046575] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:20 2019][2937782.051365] .tg->runnable_avg : 0 [Fri Apr 12 11:14:20 2019][2937782.056159] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:20 2019][2937782.060987] .throttled : 0 [Fri Apr 12 11:14:20 2019][2937782.065780] .throttle_count : 0 [Fri Apr 12 11:14:20 2019][2937782.070572] .avg->runnable_avg_sum : 5413 [Fri Apr 12 11:14:20 2019][2937782.075624] .avg->runnable_avg_period : 47004 [Fri Apr 12 11:14:20 2019][2937782.080771] [Fri Apr 12 11:14:20 2019][2937782.082446] rt_rq[23]:/ [Fri Apr 12 11:14:20 2019][2937782.085072] .rt_nr_running : 0 [Fri Apr 12 11:14:20 2019][2937782.089865] .rt_throttled : 0 [Fri Apr 12 11:14:20 2019][2937782.094659] .rt_time : 0.000000 [Fri Apr 12 11:14:20 2019][2937782.100059] .rt_runtime : 1000.000000 [Fri Apr 12 11:14:20 2019][2937782.105716] [Fri Apr 12 11:14:20 2019][2937782.107389] runnable tasks: [Fri Apr 12 11:14:20 2019][2937782.110361] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:20 2019][2937782.121307] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:20 2019][2937782.132193] watchdog/23 126 -11.980694 734431 0 0.000000 22572.732962 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.143528] migration/23 127 0.000000 8325 0 0.000000 72208.050149 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.154864] ksoftirqd/23 128 83624477.033723 15711962 120 0.000000 171156.074120 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.166201] kworker/23:0H 130 3877.002992 10 100 0.000000 0.036830 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.177553] khungtaskd 257 83616884.518922 24497 120 0.000000 7179.226377 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.188914] jbd2/sdk2-8 7793 83624420.679099 967902 120 0.000000 25156.867379 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.200270] ccp-6-q2 13584 2785.170508 2 120 0.000000 0.009198 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.211632] kdmwork-253:1 17187 83624559.752826 569320 120 0.000000 11799.185236 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.222978] systemd-logind 18110 83623734.788219 2853439 120 0.000000 105497.004897 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.234320] rs:main Q:Reg 19307 83616068.146941 303208 120 0.000000 15058.266622 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.245668] kworker/23:1H 20961 83624477.026338 14203261 100 0.000000 71749.092190 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.257012] kworker/23:1 23386 83624601.592323 941374 120 0.000000 14718.221300 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.268376] cfs_rh_01 117053 83623664.938347 9868 120 0.000000 6025.788931 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.279801] ptlrpc_hr03_003 117096 83624586.206077 1488308 120 0.000000 27923.736039 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.291222] ptlrpc_hr03_005 117098 83624596.692090 1487837 120 0.000000 27548.674315 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.302654] ptlrpcd_03_06 117176 83624588.726141 2293387 120 0.000000 53401.672110 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.314091] mdt_seqm_0001 117503 83578406.299689 672 120 0.000000 15.458312 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.325523] osp-pre-1-2 117565 83624435.447928 36804 120 0.000000 782.069117 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.336982] osp-pre-3-2 117567 83624458.931224 36806 120 0.000000 608.947852 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.348410] osp-pre-9-2 117589 83624517.635816 41430 120 0.000000 1072.041216 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.359830] osp-pre-43-2 117593 83624373.316774 40711 120 0.000000 926.926247 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.371256] osp-pre-44-2 117607 83624415.080427 40679 120 0.000000 956.095943 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.382685] osp-pre-23-2 117619 83624505.263728 40655 120 0.000000 981.435201 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.394106] osp-pre-20-2 117623 83624454.048662 41197 120 0.000000 980.932529 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.405541] osp-pre-47-2 117635 83624405.796132 41463 120 0.000000 924.051750 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.416990] osp-syn-1-2 117642 83624437.736150 235516 120 0.000000 6483.718018 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.428410] osp-syn-37-2 117650 83624437.689402 236229 120 0.000000 6353.123036 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.439833] osp-syn-36-2 117652 83624437.657802 235431 120 0.000000 6602.441402 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.451271] osp-syn-39-0 117724 83576106.611910 74095 120 0.000000 2275.257110 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.462697] osp-syn-31-0 117736 83574615.880929 76174 120 0.000000 2209.257077 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.474161] osp-syn-42-0 117738 83607384.435522 74816 120 0.000000 2297.014679 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.485587] osp-pre-25-0 117751 83624391.623555 39348 120 0.000000 843.974157 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.497010] osp-syn-32-0 117760 83607660.175444 74948 120 0.000000 2199.576138 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.508435] osp-syn-47-0 117786 83607926.895416 74940 120 0.000000 2201.857309 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.519855] osp-pre-1-0 117795 83624413.241322 39286 120 0.000000 853.069269 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.531277] osp-syn-37-0 117802 83584754.065757 74892 120 0.000000 2227.413863 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.542702] lquota_wb_fir-M 117836 83622037.811638 57113 120 0.000000 1876.457178 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.554139] mdt03_014 118230 83624622.673635 15764167 120 0.000000 872184.314891 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.565574] mdt03_027 118343 83624366.297814 12544493 120 0.000000 701274.240580 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.576996] mdt03_029 118357 83575379.454970 9332518 120 0.000000 731089.898201 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.588426] mdt03_032 118373 83581790.412161 20938564 120 0.000000 1226476.386148 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.599852] mdt03_038 118415 83624390.641623 8382371 120 0.000000 655070.908913 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.611299] mdt03_039 118422 83624366.298274 16936034 120 0.000000 903119.489180 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.622724] mdt03_047 118463 59696821.130796 2323227 120 0.000000 85429.843163 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.634144] mdt03_049 118466 83624420.153761 15893019 120 0.000000 827947.427964 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.645568] mdt_rdpg03_009 118500 83624366.298795 17960656 120 0.000000 767313.039148 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.657011] mdt_rdpg03_024 118544 83604815.535283 14473390 120 0.000000 580693.224980 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.668456] mdt03_054 118546 83293576.234673 16356524 120 0.000000 759174.139099 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.679886] mdt03_071 118631 83624448.739171 19000539 120 0.000000 1028091.643832 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.691317] mdt03_079 118669 83293665.741397 449125 120 0.000000 29498.376564 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.702745] mdt03_085 118699 83624420.153601 18503298 120 0.000000 1058189.786421 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.714166] mdt03_088 118712 83624390.642114 12718081 120 0.000000 825744.663364 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.725588] mdt03_089 118716 83624366.297733 21726714 120 0.000000 1207151.295701 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.737049] mdt03_096 118739 83624621.754246 8639341 120 0.000000 754811.194236 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.748469] mdt_io03_034 118740 67487403.538356 32307 120 0.000000 1251.867665 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.759902] mdt_io03_040 118770 67487403.847947 38 120 0.000000 2.442279 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.771323] mdt03_103 118776 83624624.936177 1854918 120 0.000000 120878.528611 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.782757] mdt_io03_052 118814 67487462.130062 193659 120 0.000000 7507.091896 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.794191] mdt_io03_074 118892 67487404.027306 228337 120 0.000000 8102.395976 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.805621] mdt_io03_080 118910 67487404.024963 44 120 0.000000 1.451349 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.817119] mdt_rdpg03_048 119255 83624643.030738 13376246 120 0.000000 570353.386752 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.828544] mdt_rdpg03_056 119264 83604815.534021 15881958 120 0.000000 618121.571774 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.839968] ldlm_cn03_008 119787 83624434.866089 45868610 120 0.000000 884368.731290 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.851401] ldlm_bl_08 128941 83624628.124061 700710 120 0.000000 46481.485146 0.000000 3 / [Fri Apr 12 11:14:20 2019][2937782.862828] ldlm_bl_12 128945 83624627.909155 700833 120 0.000000 45368.339874 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.874272] osp-pre-10-3 2241 83624580.485195 27759 120 0.000000 526.597274 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.885606] osp-pre-19-3 2259 83624540.948811 28043 120 0.000000 538.920972 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.896942] osp-syn-7-3 2288 83624505.127962 9393 120 0.000000 303.484063 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.908929] osp-syn-38-3 2300 83623992.219924 9467 120 0.000000 274.861831 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.920263] osp-pre-22-3 2309 83624501.342980 27847 120 0.000000 784.234734 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.931618] osp-syn-12-1 2390 83622171.077235 20299 120 0.000000 479.318201 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.942955] osp-syn-2-1 2412 83622113.307253 19621 120 0.000000 496.082091 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.954289] osp-pre-23-1 2441 83624507.200885 27566 120 0.000000 543.094818 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.965624] osp-pre-20-1 2445 83624470.680948 27854 120 0.000000 616.154788 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.976960] osp-pre-7-1 2447 83624510.674391 27845 120 0.000000 642.727406 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.988296] osp-syn-40-1 2448 83622281.101190 19834 120 0.000000 487.774610 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937782.999633] osp-syn-45-1 2450 83622238.866531 19946 120 0.000000 488.234897 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.010971] osp-pre-24-1 2455 83624564.092528 28041 120 0.000000 545.696656 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.022322] osp-syn-38-1 2458 83622063.217694 20036 120 0.000000 484.806993 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.033669] osp-pre-38-1 2459 83624631.123011 27586 120 0.000000 648.449293 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.045004] osp-syn-15-1 2460 83621763.182532 20343 120 0.000000 538.259129 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.056342] osp-syn-22-1 2466 83622017.077769 19854 120 0.000000 486.050401 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.067679] osp-syn-8-1 2476 83621984.617701 19824 120 0.000000 490.791312 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.079013] osp-pre-8-1 2477 83624507.197198 28042 120 0.000000 616.949285 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.090350] osp-syn-30-1 2482 83622240.233146 19705 120 0.000000 476.074113 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.101694] lquota_wb_fir-M 2503 83622005.752553 2315 120 0.000000 21.635035 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.113032] mdt_io03_089 3059 67487415.695106 23 120 0.000000 1.823421 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.124385] ldlm_cn03_028 45020 83624653.920945 945478 120 0.000000 26901.620305 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.135722] kworker/u256:2 45906 83602796.592642 4976 120 0.000000 95.879451 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.147057] kworker/u256:0 46263 83624600.867147 3046 120 0.000000 57.556809 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.158410] kworker/23:0 46356 83557259.745556 2 120 0.000000 0.009858 0.000000 3 / [Fri Apr 12 11:14:21 2019][2937783.169809] [Fri Apr 12 11:14:21 2019][2937783.171484] cpu#24, 1996.203 MHz [Fri Apr 12 11:14:21 2019][2937783.174908] .nr_running : 0 [Fri Apr 12 11:14:21 2019][2937783.179711] .load : 0 [Fri Apr 12 11:14:21 2019][2937783.184512] .nr_switches : 20975744080 [Fri Apr 12 11:14:21 2019][2937783.190181] .nr_load_updates : 1915732484 [Fri Apr 12 11:14:21 2019][2937783.195763] .nr_uninterruptible : -558792 [Fri Apr 12 11:14:21 2019][2937783.201083] .next_balance : 7232.384064 [Fri Apr 12 11:14:21 2019][2937783.206755] .curr->pid : 0 [Fri Apr 12 11:14:21 2019][2937783.211568] .clock : 2937783211.566866 [Fri Apr 12 11:14:21 2019][2937783.217749] .cpu_load[0] : 0 [Fri Apr 12 11:14:21 2019][2937783.222552] .cpu_load[1] : 0 [Fri Apr 12 11:14:21 2019][2937783.227352] .cpu_load[2] : 261 [Fri Apr 12 11:14:21 2019][2937783.232327] .cpu_load[3] : 211 [Fri Apr 12 11:14:21 2019][2937783.237300] .cpu_load[4] : 226 [Fri Apr 12 11:14:21 2019][2937783.242268] .avg_idle : 153798 [Fri Apr 12 11:14:21 2019][2937783.247502] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:21 2019][2937783.252823] [Fri Apr 12 11:14:21 2019][2937783.254492] cfs_rq[24]:/ [Fri Apr 12 11:14:21 2019][2937783.257205] .exec_clock : 0.000000 [Fri Apr 12 11:14:21 2019][2937783.262605] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:21 2019][2937783.268004] .min_vruntime : 112597244.396397 [Fri Apr 12 11:14:21 2019][2937783.274096] .max_vruntime : 0.000001 [Fri Apr 12 11:14:21 2019][2937783.279497] .spread : 0.000000 [Fri Apr 12 11:14:21 2019][2937783.284896] .spread0 : -4246285.018119 [Fri Apr 12 11:14:21 2019][2937783.290963] .nr_spread_over : 0 [Fri Apr 12 11:14:21 2019][2937783.295756] .nr_running : 0 [Fri Apr 12 11:14:21 2019][2937783.300600] .load : 0 [Fri Apr 12 11:14:21 2019][2937783.305393] .runnable_load_avg : 0 [Fri Apr 12 11:14:21 2019][2937783.310195] .blocked_load_avg : 541 [Fri Apr 12 11:14:21 2019][2937783.315160] .tg_load_avg : 0 [Fri Apr 12 11:14:21 2019][2937783.320022] .tg_load_contrib : 0 [Fri Apr 12 11:14:21 2019][2937783.324898] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:21 2019][2937783.329759] .tg->runnable_avg : 0 [Fri Apr 12 11:14:21 2019][2937783.334574] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:21 2019][2937783.339367] .throttled : 0 [Fri Apr 12 11:14:21 2019][2937783.344160] .throttle_count : 0 [Fri Apr 12 11:14:21 2019][2937783.348952] .avg->runnable_avg_sum : 7505 [Fri Apr 12 11:14:21 2019][2937783.354005] .avg->runnable_avg_period : 46247 [Fri Apr 12 11:14:21 2019][2937783.359151] [Fri Apr 12 11:14:21 2019][2937783.360825] rt_rq[24]:/ [Fri Apr 12 11:14:21 2019][2937783.363451] .rt_nr_running : 0 [Fri Apr 12 11:14:21 2019][2937783.368244] .rt_throttled : 0 [Fri Apr 12 11:14:21 2019][2937783.373037] .rt_time : 0.009318 [Fri Apr 12 11:14:21 2019][2937783.378437] .rt_runtime : 950.000000 [Fri Apr 12 11:14:21 2019][2937783.384008] [Fri Apr 12 11:14:21 2019][2937783.385681] runnable tasks: [Fri Apr 12 11:14:21 2019][2937783.388654] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:21 2019][2937783.399610] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:21 2019][2937783.410484] watchdog/24 131 -11.980713 734432 0 0.000000 25478.353310 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.421823] migration/24 132 0.000000 14863 0 0.000000 199100.608471 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.433158] ksoftirqd/24 133 112597239.535180 16890673 120 0.000000 252980.144990 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.444505] kworker/24:0H 135 3557.323275 9 100 0.000000 0.037654 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.455878] kworker/24:1H 16841 112597239.531573 13929928 100 0.000000 95757.371782 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.467319] dsm_sa_eventmgr 18040 112597081.608408 944896 120 0.000000 21508.124535 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.478830] kworker/24:2 102506 112597231.730112 490665 120 0.000000 11339.771698 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.490327] ptlrpcd_00_09 117143 112596887.885093 881453 120 0.000000 22639.571565 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.501755] ldlm_cn00_002 117268 112596694.394091 45798955 120 0.000000 1061893.116049 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.513217] ldlm_cn00_004 117686 112597232.612718 46385628 120 0.000000 1081057.850844 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.524686] mdt00_034 118233 112597083.532713 22231754 120 0.000000 1474296.116266 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.536114] mdt00_042 118259 112590734.236579 23288803 120 0.000000 1351049.730902 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.547546] mdt00_063 118363 112597262.849688 27105621 120 0.000000 1388843.162353 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.558972] mdt00_076 118426 112592951.734530 21980846 120 0.000000 1362563.150117 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.570399] mdt_io00_006 118443 97790279.834980 536245 120 0.000000 22536.285171 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.581831] mdt00_099 118553 112591207.279247 15551760 120 0.000000 1127261.496769 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.593253] mdt00_100 118554 112597083.423585 28920182 120 0.000000 1608872.445555 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.604678] mdt00_109 118569 112591207.276091 34859331 120 0.000000 1778832.844440 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.616109] mdt00_111 118573 112591207.279358 29749414 120 0.000000 1757063.103991 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.627640] mdt_io00_031 118695 97790279.980081 79 120 0.000000 3.357919 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.639297] mdt_io00_061 118871 97790279.412696 66 120 0.000000 3.731419 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.650803] mdt_io00_063 118879 112597083.490873 3283693 120 0.000000 232435.104244 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.662244] mdt_io00_090 119007 97790279.424337 382450 120 0.000000 15105.963961 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.673664] mdt_io00_091 119008 97790279.400602 451582 120 0.000000 20222.025231 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.685101] mdt_rdpg00_013 119037 112597056.417423 23482437 120 0.000000 1040683.096433 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.696546] mdt_rdpg00_042 119196 112597056.408827 21293903 120 0.000000 937498.493365 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.707966] mdt_rdpg00_043 119198 112597056.413896 21637853 120 0.000000 961473.070009 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.719395] mdt_rdpg00_050 119228 112597056.414007 23269928 120 0.000000 1080335.287040 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.730830] ldlm_cn00_012 123637 112597269.830433 44235698 120 0.000000 1053876.363011 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.742255] ldlm_cn00_013 123639 112596694.394372 41797990 120 0.000000 1010090.024213 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.753682] ldlm_cn00_015 126538 112597232.398691 35531865 120 0.000000 880720.946036 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.765115] ldlm_cn00_023 129947 112597279.825143 34792386 120 0.000000 862647.916918 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.776579] osp-pre-13-3 2235 112597239.504452 28038 120 0.000000 586.716834 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.788095] ldlm_cn00_026 6483 112597232.458315 30200925 120 0.000000 752911.075711 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.799559] ldlm_cn00_027 6484 112596665.391358 27503384 120 0.000000 707209.717871 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.810905] ldlm_cn00_031 6488 112596694.394060 29621713 120 0.000000 727692.668560 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.822251] kworker/24:0 43310 110718820.502217 2 120 0.000000 0.009087 0.000000 0 / [Fri Apr 12 11:14:21 2019][2937783.833602] [Fri Apr 12 11:14:21 2019][2937783.835280] cpu#25, 1996.203 MHz [Fri Apr 12 11:14:21 2019][2937783.838695] .nr_running : 1 [Fri Apr 12 11:14:21 2019][2937783.843499] .load : 1024 [Fri Apr 12 11:14:21 2019][2937783.848593] .nr_switches : 22152616588 [Fri Apr 12 11:14:21 2019][2937783.854262] .nr_load_updates : 2004768691 [Fri Apr 12 11:14:21 2019][2937783.859846] .nr_uninterruptible : 71907 [Fri Apr 12 11:14:21 2019][2937783.864990] .next_balance : 7232.384609 [Fri Apr 12 11:14:22 2019][2937783.870661] .curr->pid : 0 [Fri Apr 12 11:14:22 2019][2937783.875462] .clock : 2937783875.446251 [Fri Apr 12 11:14:22 2019][2937783.881648] .cpu_load[0] : 0 [Fri Apr 12 11:14:22 2019][2937783.886450] .cpu_load[1] : 516 [Fri Apr 12 11:14:22 2019][2937783.891426] .cpu_load[2] : 676 [Fri Apr 12 11:14:22 2019][2937783.896402] .cpu_load[3] : 635 [Fri Apr 12 11:14:22 2019][2937783.901374] .cpu_load[4] : 453 [Fri Apr 12 11:14:22 2019][2937783.906351] .avg_idle : 46419 [Fri Apr 12 11:14:22 2019][2937783.911499] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:22 2019][2937783.916820] [Fri Apr 12 11:14:22 2019][2937783.918496] cfs_rq[25]:/ [Fri Apr 12 11:14:22 2019][2937783.921209] .exec_clock : 0.000000 [Fri Apr 12 11:14:22 2019][2937783.926608] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:22 2019][2937783.932007] .min_vruntime : 128675600.035231 [Fri Apr 12 11:14:22 2019][2937783.938099] .max_vruntime : 0.000001 [Fri Apr 12 11:14:22 2019][2937783.943498] .spread : 0.000000 [Fri Apr 12 11:14:22 2019][2937783.948898] .spread0 : 11831967.673101 [Fri Apr 12 11:14:22 2019][2937783.954904] .nr_spread_over : 0 [Fri Apr 12 11:14:22 2019][2937783.959697] .nr_running : 0 [Fri Apr 12 11:14:22 2019][2937783.964491] .load : 0 [Fri Apr 12 11:14:22 2019][2937783.969284] .runnable_load_avg : 90 [Fri Apr 12 11:14:22 2019][2937783.974172] .blocked_load_avg : 0 [Fri Apr 12 11:14:22 2019][2937783.978965] .tg_load_avg : 0 [Fri Apr 12 11:14:22 2019][2937783.983756] .tg_load_contrib : 0 [Fri Apr 12 11:14:22 2019][2937783.988549] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:22 2019][2937783.993343] .tg->runnable_avg : 0 [Fri Apr 12 11:14:22 2019][2937783.998135] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:22 2019][2937784.002928] .throttled : 0 [Fri Apr 12 11:14:22 2019][2937784.007720] .throttle_count : 0 [Fri Apr 12 11:14:22 2019][2937784.012513] .avg->runnable_avg_sum : 27590 [Fri Apr 12 11:14:22 2019][2937784.017652] .avg->runnable_avg_period : 46104 [Fri Apr 12 11:14:22 2019][2937784.022801] [Fri Apr 12 11:14:22 2019][2937784.024474] rt_rq[25]:/ [Fri Apr 12 11:14:22 2019][2937784.027099] .rt_nr_running : 0 [Fri Apr 12 11:14:22 2019][2937784.031902] .rt_throttled : 0 [Fri Apr 12 11:14:22 2019][2937784.036712] .rt_time : 0.000000 [Fri Apr 12 11:14:22 2019][2937784.042120] .rt_runtime : 950.000000 [Fri Apr 12 11:14:22 2019][2937784.047691] [Fri Apr 12 11:14:22 2019][2937784.049364] runnable tasks: [Fri Apr 12 11:14:22 2019][2937784.052338] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:22 2019][2937784.063283] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:22 2019][2937784.074160] watchdog/25 136 -11.985081 734432 0 0.000000 37457.547131 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.085496] migration/25 137 0.000000 24586 0 0.000000 335552.166619 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.096833] ksoftirqd/25 138 128675474.429016 12564325 120 0.000000 293623.503958 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.108170] kworker/25:0H 140 4361.226517 10 100 0.000000 0.039234 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.119534] ccp-4-q0 13534 3221.153873 2 120 0.000000 0.011473 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.130867] ccp-4-q1 13539 3233.162985 2 120 0.000000 0.009118 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.142207] ccp-7-q2 13776 3350.147708 2 120 0.000000 0.011021 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.153539] ccp-7-q3 13777 3362.157832 2 120 0.000000 0.010130 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.164884] ccp-7-q4 13778 3374.167625 2 120 0.000000 0.009799 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.176256] kworker/25:1H 20553 128675474.412159 8328764 100 0.000000 66945.402330 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.187624] kworker/25:2 85417 128675655.902854 1557899 120 0.000000 54850.831586 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.198967] kworker/25:1 96912 91658091.718803 2 120 0.000000 0.009568 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.210317] ldlm_cb01_000 117259 128561259.631903 626 120 0.000000 33.013559 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.221752] mdt_io01_000 117510 118500476.262928 97 120 0.000000 3.919482 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.233196] ldlm_cn01_005 117687 128675202.339279 51689788 120 0.000000 1295037.978762 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.244647] mdt01_010 118149 123073524.364445 17960132 120 0.000000 1116414.718352 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.256081] mdt01_032 118225 128612146.604913 25536070 120 0.000000 1711423.006148 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.267510] mdt01_073 118335 128594586.290541 23817383 120 0.000000 1663507.706238 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.278953] mdt_out01_003 118595 128561203.252186 37 120 0.000000 0.268515 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.290382] mdt_out01_005 118676 128561203.507711 99326 120 0.000000 4278.614682 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.301829] mdt_io01_046 118880 118500476.261797 1950538 120 0.000000 94087.307352 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.313278] mdt_rdpg01_010 119042 128675694.408642 31544759 120 0.000000 1423361.507467 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.324702] mdt_io01_074 119044 118500476.263670 471657 120 0.000000 17651.402472 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.336128] mdt_io01_079 119060 118500476.262849 922336 120 0.000000 50110.443611 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.347555] mdt_rdpg01_027 119102 128675701.415428 27012406 120 0.000000 1291829.055373 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.358991] mdt_io01_099 119162 118500476.262468 35136 120 0.000000 1666.495771 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.370411] mdt_io01_100 119169 118500476.262539 1318687 120 0.000000 72744.310478 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.381840] mdt_io01_108 119215 118500833.740008 511592 120 0.000000 45465.658169 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.393285] ldlm_cn01_008 119542 129033611.173102 50987008 120 0.000000 1268435.663933 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.404708] ldlm_cn01_010 121244 128675202.339189 48509212 120 0.000000 1238680.973267 0.000000 1 / [Fri Apr 12 11:14:22 2019][2937784.416193] [Fri Apr 12 11:14:22 2019][2937784.417866] cpu#26, 1996.203 MHz [Fri Apr 12 11:14:22 2019][2937784.421272] .nr_running : 0 [Fri Apr 12 11:14:22 2019][2937784.426066] .load : 1024 [Fri Apr 12 11:14:22 2019][2937784.431128] .nr_switches : 30958729597 [Fri Apr 12 11:14:22 2019][2937784.436797] .nr_load_updates : 1908331865 [Fri Apr 12 11:14:22 2019][2937784.442380] .nr_uninterruptible : 17604 [Fri Apr 12 11:14:22 2019][2937784.447524] .next_balance : 7232.385194 [Fri Apr 12 11:14:22 2019][2937784.453184] .curr->pid : 0 [Fri Apr 12 11:14:22 2019][2937784.457985] .clock : 2937784457.901953 [Fri Apr 12 11:14:22 2019][2937784.464166] .cpu_load[0] : 2048 [Fri Apr 12 11:14:22 2019][2937784.469225] .cpu_load[1] : 285 [Fri Apr 12 11:14:22 2019][2937784.474191] .cpu_load[2] : 1463 [Fri Apr 12 11:14:22 2019][2937784.479246] .cpu_load[3] : 1038 [Fri Apr 12 11:14:22 2019][2937784.484329] .cpu_load[4] : 748 [Fri Apr 12 11:14:22 2019][2937784.489310] .avg_idle : 44530 [Fri Apr 12 11:14:22 2019][2937784.494457] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:22 2019][2937784.499782] [Fri Apr 12 11:14:22 2019][2937784.501463] cfs_rq[26]:/ [Fri Apr 12 11:14:22 2019][2937784.504178] .exec_clock : 0.000000 [Fri Apr 12 11:14:22 2019][2937784.509575] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:22 2019][2937784.514986] .min_vruntime : 138966196.230691 [Fri Apr 12 11:14:22 2019][2937784.521078] .max_vruntime : 0.000001 [Fri Apr 12 11:14:22 2019][2937784.526475] .spread : 0.000000 [Fri Apr 12 11:14:22 2019][2937784.531877] .spread0 : 22122494.746923 [Fri Apr 12 11:14:22 2019][2937784.537890] .nr_spread_over : 0 [Fri Apr 12 11:14:22 2019][2937784.542684] .nr_running : 1 [Fri Apr 12 11:14:22 2019][2937784.547476] .load : 0 [Fri Apr 12 11:14:22 2019][2937784.552268] .runnable_load_avg : 261 [Fri Apr 12 11:14:22 2019][2937784.557236] .blocked_load_avg : 974 [Fri Apr 12 11:14:22 2019][2937784.562201] .tg_load_avg : 0 [Fri Apr 12 11:14:22 2019][2937784.566995] .tg_load_contrib : 0 [Fri Apr 12 11:14:22 2019][2937784.571785] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:22 2019][2937784.576580] .tg->runnable_avg : 0 [Fri Apr 12 11:14:22 2019][2937784.581372] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:22 2019][2937784.586165] .throttled : 0 [Fri Apr 12 11:14:22 2019][2937784.590958] .throttle_count : 0 [Fri Apr 12 11:14:22 2019][2937784.595758] .avg->runnable_avg_sum : 12228 [Fri Apr 12 11:14:22 2019][2937784.600898] .avg->runnable_avg_period : 46762 [Fri Apr 12 11:14:22 2019][2937784.606045] [Fri Apr 12 11:14:22 2019][2937784.607718] rt_rq[26]:/ [Fri Apr 12 11:14:22 2019][2937784.610344] .rt_nr_running : 0 [Fri Apr 12 11:14:22 2019][2937784.615145] .rt_throttled : 0 [Fri Apr 12 11:14:22 2019][2937784.619940] .rt_time : 0.000000 [Fri Apr 12 11:14:22 2019][2937784.625340] .rt_runtime : 950.000000 [Fri Apr 12 11:14:22 2019][2937784.630910] [Fri Apr 12 11:14:22 2019][2937784.632584] runnable tasks: [Fri Apr 12 11:14:22 2019][2937784.635556] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:22 2019][2937784.646503] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:22 2019][2937784.657383] watchdog/26 141 -11.986504 734432 0 0.000000 29195.364760 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.668716] migration/26 142 0.000000 21745 0 0.000000 322401.573442 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.680051] ksoftirqd/26 143 138966208.105597 34570433 120 0.000000 989340.778709 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.691390] kworker/26:0H 145 5137.255200 10 100 0.000000 0.037715 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.702745] kworker/26:1 7304 138966127.245749 2558428 120 0.000000 63327.420594 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.714086] ccp-4-q2 13540 3641.967174 2 120 0.000000 0.009759 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.725425] ccp-6-q0 13580 3666.869956 2 120 0.000000 0.010290 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.736786] kworker/26:1H 20543 138963759.541230 3753209 100 0.000000 19877.193869 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.748134] kworker/26:0 109637 99734421.087001 2 120 0.000000 0.006203 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.759587] kiblnd_sd_02_00 117123 69696306.236650 1947272408 120 0.000000 16178375.452907 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.771097] kiblnd_sd_02_01 117124 96828141.993984 1947396198 120 0.000000 16168831.285567 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.782608] kiblnd_sd_02_02 117125 77852958.256689 1947341441 120 0.000000 16165470.675520 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.794129] ptlrpcd_02_01 117159 138965608.979614 855943 120 0.000000 21499.345091 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.805577] mdt02_007 117536 138965508.590655 18250801 120 0.000000 1186067.779534 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.817054] mdt02_017 118192 138920418.415400 16739963 120 0.000000 1262158.028976 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.828508] mdt02_057 118442 138964399.921479 22545103 120 0.000000 1450641.718078 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.839936] mdt02_076 118503 138963593.777490 13452885 120 0.000000 974477.453971 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.851376] mdt02_110 118591 138918059.423812 13522509 120 0.000000 1022881.924110 0.000000 2 / [Fri Apr 12 11:14:22 2019][2937784.862813] mdt_io02_028 118681 119708961.040718 125023 120 0.000000 6193.849094 0.000000 2 / [Fri Apr 12 11:14:23 2019][2937784.874255] mdt_io02_071 118852 119708931.305413 381581 120 0.000000 17794.549371 0.000000 2 / [Fri Apr 12 11:14:23 2019][2937784.885682] mdt_io02_072 118856 119708697.095498 34 120 0.000000 2.044113 0.000000 2 / [Fri Apr 12 11:14:23 2019][2937784.897111] mdt_io02_080 118891 119708700.710719 56 120 0.000000 2.538387 0.000000 2 / [Fri Apr 12 11:14:23 2019][2937784.908562] mdt_rdpg02_013 119054 138965493.754983 21370595 120 0.000000 1015391.403459 0.000000 2 / [Fri Apr 12 11:14:23 2019][2937784.920000] mdt_rdpg02_029 119121 138965493.754944 27646588 120 0.000000 1287222.461404 0.000000 2 / [Fri Apr 12 11:14:23 2019][2937784.931446] mdt_rdpg02_059 119249 138965493.755033 26300615 120 0.000000 1174959.120231 0.000000 2 / [Fri Apr 12 11:14:23 2019][2937784.942912] mdt_out02_003 2516 138788936.820230 14 120 0.000000 0.214729 0.000000 2 / [Fri Apr 12 11:14:23 2019][2937784.954577] [Fri Apr 12 11:14:23 2019][2937784.956253] cpu#27, 1996.203 MHz [Fri Apr 12 11:14:23 2019][2937784.959677] .nr_running : 0 [Fri Apr 12 11:14:23 2019][2937784.964478] .load : 0 [Fri Apr 12 11:14:23 2019][2937784.969278] .nr_switches : 9537369153 [Fri Apr 12 11:14:23 2019][2937784.974854] .nr_load_updates : 1551356346 [Fri Apr 12 11:14:23 2019][2937784.980433] .nr_uninterruptible : 29817 [Fri Apr 12 11:14:23 2019][2937784.985582] .next_balance : 7232.385730 [Fri Apr 12 11:14:23 2019][2937784.991250] .curr->pid : 0 [Fri Apr 12 11:14:23 2019][2937784.996051] .clock : 2937784995.805132 [Fri Apr 12 11:14:23 2019][2937785.002249] .cpu_load[0] : 1024 [Fri Apr 12 11:14:23 2019][2937785.007310] .cpu_load[1] : 16 [Fri Apr 12 11:14:23 2019][2937785.012207] .cpu_load[2] : 13 [Fri Apr 12 11:14:23 2019][2937785.017094] .cpu_load[3] : 146 [Fri Apr 12 11:14:23 2019][2937785.022068] .cpu_load[4] : 75 [Fri Apr 12 11:14:23 2019][2937785.026957] .avg_idle : 219617 [Fri Apr 12 11:14:23 2019][2937785.032192] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:23 2019][2937785.037515] [Fri Apr 12 11:14:23 2019][2937785.039192] cfs_rq[27]:/ [Fri Apr 12 11:14:23 2019][2937785.041902] .exec_clock : 0.000000 [Fri Apr 12 11:14:23 2019][2937785.047302] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:23 2019][2937785.052702] .min_vruntime : 61765133.294576 [Fri Apr 12 11:14:23 2019][2937785.058708] .max_vruntime : 0.000001 [Fri Apr 12 11:14:23 2019][2937785.064107] .spread : 0.000000 [Fri Apr 12 11:14:23 2019][2937785.069508] .spread0 : -55078697.700962 [Fri Apr 12 11:14:23 2019][2937785.075600] .nr_spread_over : 0 [Fri Apr 12 11:14:23 2019][2937785.080392] .nr_running : 0 [Fri Apr 12 11:14:23 2019][2937785.085184] .load : 0 [Fri Apr 12 11:14:23 2019][2937785.089978] .runnable_load_avg : 148 [Fri Apr 12 11:14:23 2019][2937785.094944] .blocked_load_avg : 194 [Fri Apr 12 11:14:23 2019][2937785.099911] .tg_load_avg : 0 [Fri Apr 12 11:14:23 2019][2937785.104701] .tg_load_contrib : 0 [Fri Apr 12 11:14:23 2019][2937785.109496] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:23 2019][2937785.114288] .tg->runnable_avg : 0 [Fri Apr 12 11:14:23 2019][2937785.119080] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:23 2019][2937785.123873] .throttled : 0 [Fri Apr 12 11:14:23 2019][2937785.128667] .throttle_count : 0 [Fri Apr 12 11:14:23 2019][2937785.133461] .avg->runnable_avg_sum : 4237 [Fri Apr 12 11:14:23 2019][2937785.138512] .avg->runnable_avg_period : 46487 [Fri Apr 12 11:14:23 2019][2937785.143659] [Fri Apr 12 11:14:23 2019][2937785.145334] rt_rq[27]:/ [Fri Apr 12 11:14:23 2019][2937785.147960] .rt_nr_running : 0 [Fri Apr 12 11:14:23 2019][2937785.152753] .rt_throttled : 0 [Fri Apr 12 11:14:23 2019][2937785.157545] .rt_time : 0.000000 [Fri Apr 12 11:14:23 2019][2937785.162945] .rt_runtime : 950.000000 [Fri Apr 12 11:14:23 2019][2937785.168518] [Fri Apr 12 11:14:23 2019][2937785.170188] runnable tasks: [Fri Apr 12 11:14:23 2019][2937785.173162] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:23 2019][2937785.184109] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:23 2019][2937785.194988] watchdog/27 146 -11.985072 734432 0 0.000000 10164.894978 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.206322] migration/27 147 0.000000 6353 0 0.000000 37669.308913 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.217658] ksoftirqd/27 148 61765121.315075 4525358 120 0.000000 66936.392376 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.228993] kworker/27:0H 150 4196.310875 10 100 0.000000 0.044254 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.240364] kmpathd 14017 3059.190934 2 100 0.000000 0.015048 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.251726] kworker/27:1H 20554 61765121.294609 2989811 100 0.000000 14558.854182 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.263078] kworker/27:2 1805 61765135.866235 287647 120 0.000000 5903.112150 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.274425] kiblnd_sd_03_03 117130 61765149.614408 1411218307 120 0.000000 11220819.141109 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.285940] ptlrpcd_03_10 117180 61765120.632731 2292976 120 0.000000 53724.193450 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.297370] mdt_fld_0001 117505 61596021.924100 411 120 0.000000 6.965191 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.308913] osp-syn-9-2 117588 61765120.462575 236662 120 0.000000 7047.707735 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.320337] osp-pre-25-2 117599 61765008.499664 40264 120 0.000000 996.884395 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.331761] osp-pre-32-2 117609 61764870.322885 40365 120 0.000000 998.175504 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.343203] osp_up2-0 117713 61735515.297439 522 120 0.000000 26.675021 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.355417] osp-syn-10-0 117728 61732133.937682 75220 120 0.000000 2350.456012 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.366931] osp-pre-19-0 117747 61764809.106377 39373 120 0.000000 954.657541 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.378390] osp-pre-4-0 117769 61764846.597868 40594 120 0.000000 856.656939 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.389819] osp-syn-38-0 117788 61751835.535697 74870 120 0.000000 2213.110679 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.401242] osp-syn-8-0 117806 61737693.348169 75628 120 0.000000 2344.800233 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.412672] mdt03_008 118166 61764722.987257 11015528 120 0.000000 869924.432301 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.424108] mdt03_020 118268 61764595.437976 1778591 120 0.000000 117546.559894 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.435552] mdt03_044 118454 61725355.261132 16727111 120 0.000000 986394.682303 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.446979] mdt_rdpg03_008 118498 61760567.447198 14588699 120 0.000000 642372.942777 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.458410] mdt_rdpg03_021 118538 61764595.371982 10008498 120 0.000000 394323.337126 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.469828] mdt03_052 118543 50343443.990311 784420 120 0.000000 28098.649372 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.481252] mdt03_057 118556 61725355.256825 6377728 120 0.000000 285812.455146 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.492709] mdt_io03_022 118693 50856081.009869 57 120 0.000000 3.042832 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.504135] mdt_io03_026 118710 50856081.668923 38 120 0.000000 2.148252 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.515618] mdt_rdpg03_031 119129 61756882.863695 15508680 120 0.000000 618568.175571 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.527044] mdt_rdpg03_034 119165 61756882.863314 17637007 120 0.000000 808146.764821 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.538484] mdt_rdpg03_058 119266 61764884.176908 14251249 120 0.000000 626815.430024 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.549907] ldlm_cb03_002 122130 59952976.656395 460 120 0.000000 15.026340 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.561337] ldlm_cn03_012 123641 61765162.063520 46407795 120 0.000000 903617.678813 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.572772] osp-pre-14-3 2245 61764857.495597 28024 120 0.000000 585.244230 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.584109] osp-syn-42-3 2250 61764563.548615 9491 120 0.000000 278.813598 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.595448] osp-syn-2-3 2254 61764814.398220 9500 120 0.000000 306.152941 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.606785] osp-pre-21-3 2275 61764872.919673 27842 120 0.000000 552.386782 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.618121] osp-pre-29-3 2305 61765147.198075 28005 120 0.000000 638.835532 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.629477] osp-syn-10-1 2398 61761469.116814 19823 120 0.000000 476.962893 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.640819] osp-pre-11-1 2401 61764974.119357 27749 120 0.000000 665.837835 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.652155] osp-pre-31-1 2407 61764897.374782 27839 120 0.000000 598.333940 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.663489] osp-syn-42-1 2408 61761008.224945 20092 120 0.000000 482.912051 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.674828] osp-syn-19-1 2416 61761291.380277 20079 120 0.000000 466.326963 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.686162] osp-pre-17-1 2423 61765030.373387 27519 120 0.000000 580.404447 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.697500] osp-pre-32-1 2431 61764829.430863 27882 120 0.000000 623.321859 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.708836] osp-pre-16-1 2435 61764920.482454 27725 120 0.000000 552.481141 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.720172] osp-pre-40-1 2449 61764876.271745 28092 120 0.000000 610.926282 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.731509] osp-syn-24-1 2454 61761700.311353 20149 120 0.000000 464.848883 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.742854] osp-pre-29-1 2463 61764859.796692 27843 120 0.000000 613.705359 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.754188] osp-pre-22-1 2467 61764921.214921 27731 120 0.000000 599.874311 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.765528] mdt_io03_093 3064 50856088.561143 7 120 0.000000 0.298626 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.776863] mdt_io03_096 3067 58166660.437745 1238487 120 0.000000 72748.948895 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.788217] kworker/27:0 38262 58886737.761113 2 120 0.000000 0.007194 0.000000 3 / [Fri Apr 12 11:14:23 2019][2937785.799636] [Fri Apr 12 11:14:23 2019][2937785.801316] cpu#28, 1996.203 MHz [Fri Apr 12 11:14:23 2019][2937785.804732] .nr_running : 0 [Fri Apr 12 11:14:23 2019][2937785.809531] .load : 0 [Fri Apr 12 11:14:23 2019][2937785.814323] .nr_switches : 15123716206 [Fri Apr 12 11:14:23 2019][2937785.819983] .nr_load_updates : 1853132872 [Fri Apr 12 11:14:23 2019][2937785.825556] .nr_uninterruptible : -699978 [Fri Apr 12 11:14:23 2019][2937785.830868] .next_balance : 7232.386579 [Fri Apr 12 11:14:23 2019][2937785.836528] .curr->pid : 0 [Fri Apr 12 11:14:23 2019][2937785.841321] .clock : 2937785841.319591 [Fri Apr 12 11:14:23 2019][2937785.847499] .cpu_load[0] : 0 [Fri Apr 12 11:14:23 2019][2937785.852294] .cpu_load[1] : 320 [Fri Apr 12 11:14:23 2019][2937785.857268] .cpu_load[2] : 156 [Fri Apr 12 11:14:24 2019][2937785.862233] .cpu_load[3] : 208 [Fri Apr 12 11:14:24 2019][2937785.867201] .cpu_load[4] : 405 [Fri Apr 12 11:14:24 2019][2937785.872177] .avg_idle : 49659 [Fri Apr 12 11:14:24 2019][2937785.877326] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:24 2019][2937785.882988] [Fri Apr 12 11:14:24 2019][2937785.884660] cfs_rq[28]:/ [Fri Apr 12 11:14:24 2019][2937785.887527] .exec_clock : 0.000000 [Fri Apr 12 11:14:24 2019][2937785.893145] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:24 2019][2937785.898554] .min_vruntime : 99828020.141036 [Fri Apr 12 11:14:24 2019][2937785.904569] .max_vruntime : 0.000001 [Fri Apr 12 11:14:24 2019][2937785.909967] .spread : 0.000000 [Fri Apr 12 11:14:24 2019][2937785.915367] .spread0 : -17015939.550545 [Fri Apr 12 11:14:24 2019][2937785.921460] .nr_spread_over : 0 [Fri Apr 12 11:14:24 2019][2937785.926253] .nr_running : 0 [Fri Apr 12 11:14:24 2019][2937785.931054] .load : 0 [Fri Apr 12 11:14:24 2019][2937785.935847] .runnable_load_avg : 178 [Fri Apr 12 11:14:24 2019][2937785.940812] .blocked_load_avg : 184 [Fri Apr 12 11:14:24 2019][2937785.945779] .tg_load_avg : 0 [Fri Apr 12 11:14:24 2019][2937785.950573] .tg_load_contrib : 0 [Fri Apr 12 11:14:24 2019][2937785.955366] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:24 2019][2937785.960156] .tg->runnable_avg : 0 [Fri Apr 12 11:14:24 2019][2937785.964949] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:24 2019][2937785.969742] .throttled : 0 [Fri Apr 12 11:14:24 2019][2937785.974537] .throttle_count : 0 [Fri Apr 12 11:14:24 2019][2937785.979329] .avg->runnable_avg_sum : 8374 [Fri Apr 12 11:14:24 2019][2937785.984381] .avg->runnable_avg_period : 46222 [Fri Apr 12 11:14:24 2019][2937785.989528] [Fri Apr 12 11:14:24 2019][2937785.991204] rt_rq[28]:/ [Fri Apr 12 11:14:24 2019][2937785.993829] .rt_nr_running : 0 [Fri Apr 12 11:14:24 2019][2937785.998622] .rt_throttled : 0 [Fri Apr 12 11:14:24 2019][2937786.003414] .rt_time : 0.000000 [Fri Apr 12 11:14:24 2019][2937786.008821] .rt_runtime : 950.000000 [Fri Apr 12 11:14:24 2019][2937786.014397] [Fri Apr 12 11:14:24 2019][2937786.016067] runnable tasks: [Fri Apr 12 11:14:24 2019][2937786.019039] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:24 2019][2937786.029996] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:24 2019][2937786.040871] watchdog/28 151 -11.987957 734432 0 0.000000 26111.526252 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.052207] migration/28 152 0.000000 17973 0 0.000000 227687.556777 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.063543] ksoftirqd/28 153 99828030.041819 21259500 120 0.000000 300022.015123 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.074881] kworker/28:0H 155 3497.240773 10 100 0.000000 0.052479 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.086294] kworker/28:1H 21363 99828034.107124 18090617 100 0.000000 130871.684323 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.097644] kiblnd_sd_00_01 117116 99828047.963045 2269006003 120 0.000000 19919118.316994 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.109163] mdt00_002 117465 99827318.427035 15902543 120 0.000000 907154.382312 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.120595] mdt00_003 117491 99759869.622837 30095195 120 0.000000 1686269.692798 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.132042] mdt_io00_000 117507 99783294.320738 2205666 120 0.000000 171430.886165 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.143462] mdt_io00_002 117509 85395704.118015 72 120 0.000000 3.970973 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.154889] mdt00_004 117550 99776191.340610 32020246 120 0.000000 1594750.533507 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.166366] mdt_rdpg00_004 118087 99820955.398665 24720219 120 0.000000 1166208.468285 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.177799] mdt00_027 118213 99745649.647506 29128184 120 0.000000 1616239.074510 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.189244] mdt00_080 118440 96113775.229147 17907281 120 0.000000 1054560.811864 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.200670] mdt00_106 118563 99818052.878905 27382766 120 0.000000 1511228.024369 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.212096] mdt_io00_012 118617 99783294.320989 4495899 120 0.000000 338417.437061 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.223519] mdt_io00_016 118627 85395988.273530 162 120 0.000000 10.365741 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.234942] mdt_io00_021 118647 85395704.254740 40 120 0.000000 1.581385 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.246372] mdt_io00_037 118747 99828021.062523 2644695 120 0.000000 213386.989978 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.257794] mdt_io00_038 118751 85395705.785619 70 120 0.000000 3.272105 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.269226] mdt_io00_051 118832 85395705.863486 73 120 0.000000 3.886642 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.280651] mdt_io00_054 118848 99783294.321440 965516 120 0.000000 87314.024887 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.292074] mdt_io00_055 118853 99783294.321700 2852079 120 0.000000 240133.218749 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.303505] mdt_io00_068 118906 99783294.333142 74 120 0.000000 3.069697 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.314944] mdt_io00_078 118955 99820955.656116 1228544 120 0.000000 102388.224053 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.326371] mdt_io00_086 118981 99783294.320729 181389 120 0.000000 12984.955866 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.337804] mdt_io00_096 119021 99783294.320688 3520812 120 0.000000 230598.221680 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.349241] mdt_io00_098 119031 85395984.296823 567302 120 0.000000 24288.456902 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.360667] mdt_io00_103 119059 85395988.326190 136 120 0.000000 7.513106 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.372133] ldlm_cb00_002 122745 99735433.479987 990 120 0.000000 46.317349 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.383593] kworker/28:2 5813 99828057.870811 90787 120 0.000000 5789.984335 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.394938] kworker/28:1 33122 94839476.656174 2 120 0.000000 0.008516 0.000000 0 / [Fri Apr 12 11:14:24 2019][2937786.406362] [Fri Apr 12 11:14:24 2019][2937786.408040] cpu#29, 1996.203 MHz [Fri Apr 12 11:14:24 2019][2937786.411455] .nr_running : 1 [Fri Apr 12 11:14:24 2019][2937786.416257] .load : 0 [Fri Apr 12 11:14:24 2019][2937786.421058] .nr_switches : 19223486374 [Fri Apr 12 11:14:24 2019][2937786.426726] .nr_load_updates : 1960060028 [Fri Apr 12 11:14:24 2019][2937786.432341] .nr_uninterruptible : 84125 [Fri Apr 12 11:14:24 2019][2937786.437490] .next_balance : 7232.387183 [Fri Apr 12 11:14:24 2019][2937786.443158] .curr->pid : 119183 [Fri Apr 12 11:14:24 2019][2937786.448391] .clock : 2937786448.382790 [Fri Apr 12 11:14:24 2019][2937786.454570] .cpu_load[0] : 0 [Fri Apr 12 11:14:24 2019][2937786.459385] .cpu_load[1] : 0 [Fri Apr 12 11:14:24 2019][2937786.464184] .cpu_load[2] : 446 [Fri Apr 12 11:14:24 2019][2937786.469158] .cpu_load[3] : 163 [Fri Apr 12 11:14:24 2019][2937786.474122] .cpu_load[4] : 135 [Fri Apr 12 11:14:24 2019][2937786.479090] .avg_idle : 199711 [Fri Apr 12 11:14:24 2019][2937786.484329] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:24 2019][2937786.489645] [Fri Apr 12 11:14:24 2019][2937786.491315] cfs_rq[29]:/ [Fri Apr 12 11:14:24 2019][2937786.494028] .exec_clock : 0.000000 [Fri Apr 12 11:14:24 2019][2937786.499428] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:24 2019][2937786.504825] .min_vruntime : 123358764.402253 [Fri Apr 12 11:14:24 2019][2937786.510918] .max_vruntime : 0.000001 [Fri Apr 12 11:14:24 2019][2937786.516319] .spread : 0.000000 [Fri Apr 12 11:14:24 2019][2937786.521719] .spread0 : 6514742.827152 [Fri Apr 12 11:14:24 2019][2937786.527636] .nr_spread_over : 0 [Fri Apr 12 11:14:24 2019][2937786.532431] .nr_running : 0 [Fri Apr 12 11:14:24 2019][2937786.537222] .load : 0 [Fri Apr 12 11:14:24 2019][2937786.542017] .runnable_load_avg : 0 [Fri Apr 12 11:14:24 2019][2937786.546810] .blocked_load_avg : 108 [Fri Apr 12 11:14:24 2019][2937786.551775] .tg_load_avg : 0 [Fri Apr 12 11:14:24 2019][2937786.556567] .tg_load_contrib : 0 [Fri Apr 12 11:14:24 2019][2937786.561362] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:24 2019][2937786.566152] .tg->runnable_avg : 0 [Fri Apr 12 11:14:24 2019][2937786.570945] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:24 2019][2937786.575739] .throttled : 0 [Fri Apr 12 11:14:24 2019][2937786.580532] .throttle_count : 0 [Fri Apr 12 11:14:24 2019][2937786.585324] .avg->runnable_avg_sum : 7752 [Fri Apr 12 11:14:24 2019][2937786.590376] .avg->runnable_avg_period : 46869 [Fri Apr 12 11:14:24 2019][2937786.595526] [Fri Apr 12 11:14:24 2019][2937786.597197] rt_rq[29]:/ [Fri Apr 12 11:14:24 2019][2937786.599824] .rt_nr_running : 0 [Fri Apr 12 11:14:24 2019][2937786.604617] .rt_throttled : 0 [Fri Apr 12 11:14:24 2019][2937786.609409] .rt_time : 0.000000 [Fri Apr 12 11:14:24 2019][2937786.614810] .rt_runtime : 950.000000 [Fri Apr 12 11:14:24 2019][2937786.620381] [Fri Apr 12 11:14:24 2019][2937786.622054] runnable tasks: [Fri Apr 12 11:14:24 2019][2937786.625036] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:24 2019][2937786.635982] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:24 2019][2937786.646860] watchdog/29 156 -11.986986 734432 0 0.000000 37398.108063 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.658195] migration/29 157 0.000000 27122 0 0.000000 353875.588838 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.669539] ksoftirqd/29 158 123358764.658699 14911924 120 0.000000 303157.195712 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.680884] kworker/29:0H 160 4306.484155 10 100 0.000000 0.032561 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.692239] ata_sff 3863 1059.522475 2 100 0.000000 0.012473 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.703593] multipathd 17106 0.000000 1 0 0.000000 0.565925 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.714928] multipathd 17188 0.000000 1 0 0.000000 0.570032 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.726291] multipathd 17263 0.000000 1 0 0.000000 0.907924 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.737627] multipathd 17330 0.000000 1 0 0.000000 0.593527 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.748962] multipathd 17544 0.000000 1 0 0.000000 0.941677 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.760323] kworker/29:1H 21320 123358764.637208 10193952 100 0.000000 71325.380314 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.771674] kworker/29:1 42240 123358662.169431 1407147 120 0.000000 29459.242456 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.783014] kworker/29:2 22705 105486976.683408 2 120 0.000000 0.009527 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.794372] ptlrpcd_01_03 117149 123356020.596593 674368 120 0.000000 15754.506831 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.805798] ldlm_cn01_001 117270 123358740.210105 53228630 120 0.000000 1302840.223456 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.817235] mdt01_003 117530 108533733.896187 1274917 120 0.000000 40452.940518 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.828679] ldlm_cn01_003 117681 123358092.147483 56774599 120 0.000000 1373595.623861 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.840127] mdt01_007 117898 123224683.521599 21340321 120 0.000000 1591909.688224 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.851572] mdt01_020 118182 123358322.692473 27051846 120 0.000000 1699716.537711 0.000000 1 / [Fri Apr 12 11:14:24 2019][2937786.863028] mdt01_042 118264 123247280.291049 21365039 120 0.000000 1496277.897759 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.874450] mdt01_044 118269 123358658.843410 25189275 120 0.000000 1754730.888577 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.885876] mdt01_055 118297 110019500.427541 4628314 120 0.000000 193453.224276 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.897298] mdt01_062 118317 123358796.856618 34765961 120 0.000000 2202989.753309 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.908725] mdt01_082 118349 123358791.852809 26166939 120 0.000000 1795856.657333 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.920161] mdt01_086 118353 118935131.356055 14030817 120 0.000000 864369.901364 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.931596] mdt01_098 118389 123358783.775242 26673676 120 0.000000 1812999.272020 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.943022] mdt01_101 118400 113425842.593581 4355166 120 0.000000 264097.797571 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.954461] mdt01_102 118402 109749343.978954 604100 120 0.000000 26392.959738 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.965890] mdt_io01_005 118432 123357643.830164 3559475 120 0.000000 248909.739832 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.977326] mdt01_109 118437 123358780.317923 22461500 120 0.000000 1558635.034652 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937786.988774] mdt_io01_017 118698 114347106.558449 90 120 0.000000 4.574557 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.000228] mdt_io01_033 118824 110013872.795654 157 120 0.000000 6.127562 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.011674] mdt_io01_052 118930 123196573.646770 1455919 120 0.000000 134883.182044 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.023423] mdt_io01_059 118967 110013872.449639 74 120 0.000000 3.292338 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.034854] mdt_io01_066 119005 123246314.534717 2326249 120 0.000000 205680.210325 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.046279] mdt_io01_067 119006 110013874.909313 613681 120 0.000000 20882.293306 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.057714] mdt_rdpg01_014 119061 123358557.548879 20433904 120 0.000000 1078966.064072 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.069144] mdt_io01_089 119117 123358189.738528 1648511 120 0.000000 161875.511124 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.080576] mdt_rdpg01_038 119172 123358661.152169 28455016 120 0.000000 1382747.963622 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.091999] mdt_rdpg01_046 119193 123358065.359129 22112895 120 0.000000 1219406.896841 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.103419] mdt_io01_104 119194 123358288.259968 1525310 120 0.000000 135935.236218 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.114855] ldlm_cn01_008 119542 123358816.363413 50990316 120 0.000000 1268529.259888 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.126284] ldlm_cn01_017 123805 123358740.209994 49242155 120 0.000000 1258877.542254 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.137707] ldlm_cn01_026 126536 123358340.958553 46425104 120 0.000000 1200071.376236 0.000000 1 / [Fri Apr 12 11:14:25 2019][2937787.149212] [Fri Apr 12 11:14:25 2019][2937787.150892] cpu#30, 1996.203 MHz [Fri Apr 12 11:14:25 2019][2937787.154311] .nr_running : 1 [Fri Apr 12 11:14:25 2019][2937787.159111] .load : 0 [Fri Apr 12 11:14:25 2019][2937787.163911] .nr_switches : 10759431582 [Fri Apr 12 11:14:25 2019][2937787.169581] .nr_load_updates : 1722939550 [Fri Apr 12 11:14:25 2019][2937787.175173] .nr_uninterruptible : 87886 [Fri Apr 12 11:14:25 2019][2937787.180319] .next_balance : 7232.387924 [Fri Apr 12 11:14:25 2019][2937787.185986] .curr->pid : 0 [Fri Apr 12 11:14:25 2019][2937787.190789] .clock : 2937787190.671936 [Fri Apr 12 11:14:25 2019][2937787.196994] .cpu_load[0] : 0 [Fri Apr 12 11:14:25 2019][2937787.201793] .cpu_load[1] : 1 [Fri Apr 12 11:14:25 2019][2937787.206587] .cpu_load[2] : 371 [Fri Apr 12 11:14:25 2019][2937787.211562] .cpu_load[3] : 666 [Fri Apr 12 11:14:25 2019][2937787.216557] .cpu_load[4] : 405 [Fri Apr 12 11:14:25 2019][2937787.221528] .avg_idle : 132765 [Fri Apr 12 11:14:25 2019][2937787.226756] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:25 2019][2937787.232410] [Fri Apr 12 11:14:25 2019][2937787.234400] cfs_rq[30]:/ [Fri Apr 12 11:14:25 2019][2937787.237195] .exec_clock : 0.000000 [Fri Apr 12 11:14:25 2019][2937787.242594] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:25 2019][2937787.247992] .min_vruntime : 96208298.575352 [Fri Apr 12 11:14:25 2019][2937787.253999] .max_vruntime : 0.000001 [Fri Apr 12 11:14:25 2019][2937787.259397] .spread : 0.000000 [Fri Apr 12 11:14:25 2019][2937787.264798] .spread0 : -20635819.208327 [Fri Apr 12 11:14:25 2019][2937787.270934] .nr_spread_over : 0 [Fri Apr 12 11:14:25 2019][2937787.275728] .nr_running : 0 [Fri Apr 12 11:14:25 2019][2937787.280519] .load : 0 [Fri Apr 12 11:14:25 2019][2937787.285313] .runnable_load_avg : 0 [Fri Apr 12 11:14:25 2019][2937787.290105] .blocked_load_avg : 216 [Fri Apr 12 11:14:25 2019][2937787.295071] .tg_load_avg : 0 [Fri Apr 12 11:14:25 2019][2937787.299864] .tg_load_contrib : 0 [Fri Apr 12 11:14:25 2019][2937787.304658] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:25 2019][2937787.309449] .tg->runnable_avg : 0 [Fri Apr 12 11:14:25 2019][2937787.314243] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:25 2019][2937787.319035] .throttled : 0 [Fri Apr 12 11:14:25 2019][2937787.323828] .throttle_count : 0 [Fri Apr 12 11:14:25 2019][2937787.328622] .avg->runnable_avg_sum : 6044 [Fri Apr 12 11:14:25 2019][2937787.333675] .avg->runnable_avg_period : 46296 [Fri Apr 12 11:14:25 2019][2937787.338822] [Fri Apr 12 11:14:25 2019][2937787.340494] rt_rq[30]:/ [Fri Apr 12 11:14:25 2019][2937787.343120] .rt_nr_running : 0 [Fri Apr 12 11:14:25 2019][2937787.347914] .rt_throttled : 0 [Fri Apr 12 11:14:25 2019][2937787.352706] .rt_time : 0.012825 [Fri Apr 12 11:14:25 2019][2937787.358104] .rt_runtime : 950.000000 [Fri Apr 12 11:14:25 2019][2937787.363687] [Fri Apr 12 11:14:25 2019][2937787.365359] runnable tasks: [Fri Apr 12 11:14:25 2019][2937787.368331] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:25 2019][2937787.379278] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:25 2019][2937787.390160] watchdog/30 161 -11.986785 734433 0 0.000000 36355.332369 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.401499] migration/30 162 0.000000 19602 0 0.000000 267929.507866 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.412836] ksoftirqd/30 163 96208299.589219 14539330 120 0.000000 254072.985791 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.424171] kworker/30:0H 165 4484.257876 10 100 0.000000 0.034164 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.435559] collectd 19282 96208256.914825 351497 120 0.000000 25243.147964 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.446905] kworker/30:1H 21344 96208299.539793 7761329 100 0.000000 43835.244949 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.458243] kworker/30:2 54410 96208286.582676 2073481 120 0.000000 54740.237543 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.469580] kworker/30:1 100096 41231959.036018 2 120 0.000000 0.012405 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.481019] ptlrpcd_02_04 117162 96208212.655509 835532 120 0.000000 22305.187097 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.492442] ptlrpcd_02_07 117165 96208222.225347 859929 120 0.000000 21819.224508 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.503870] ldlm_cn02_002 117274 96208106.565860 48702222 120 0.000000 1163946.745042 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.515313] osp-syn-10-2 117576 96208059.318089 236038 120 0.000000 6854.808443 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.527059] ldlm_cn02_003 117684 96207555.091443 44105333 120 0.000000 1066679.157330 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.538484] osp-pre-46-0 117727 96208150.451677 39303 120 0.000000 983.344453 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.549907] osp-pre-42-0 117739 96208229.389053 39250 120 0.000000 1001.013307 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.561339] osp-pre-32-0 117761 96207890.378085 39269 120 0.000000 947.062634 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.572779] mdt_rdpg02_003 118028 96208314.354165 19802388 120 0.000000 938973.129134 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.584219] mdt02_038 118312 96189265.036060 25246378 120 0.000000 1604006.856677 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.595645] mdt02_047 118374 96199628.223858 17789891 120 0.000000 1257284.040570 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.607083] mdt02_089 118529 96135061.284755 13674247 120 0.000000 1089109.023697 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.618517] mdt02_093 118534 96206285.884572 18660041 120 0.000000 1390399.309774 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.629944] mdt02_099 118575 85270433.841603 1839322 120 0.000000 82516.869323 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.641364] mdt02_100 118576 96206285.884742 21839254 120 0.000000 1314075.451955 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.652790] mdt_io02_008 118601 96208303.732395 3779217 120 0.000000 297694.323351 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.664210] mdt_io02_010 118609 96184491.472387 2563786 120 0.000000 222704.566811 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.675644] mdt_io02_032 118696 85240499.671406 74 120 0.000000 5.477040 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.687069] mdt_io02_043 118741 96184491.472287 103496 120 0.000000 8232.113730 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.698489] mdt_io02_051 118763 96184491.473199 1688971 120 0.000000 158335.099473 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.709916] mdt_io02_064 118815 85240728.646794 192 120 0.000000 5.835339 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.721369] mdt_io02_105 118977 85240500.401465 685822 120 0.000000 28083.789260 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.732792] mdt_io02_107 118979 85240499.904426 4159 120 0.000000 101.441688 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.744253] mdt_rdpg02_042 119168 96208324.737587 21153580 120 0.000000 1162013.305822 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.755685] mdt_io02_111 119269 96184491.471646 2914670 120 0.000000 205414.149595 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.767128] ldlm_cn02_017 129939 96208106.565830 35773419 120 0.000000 925732.431980 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.778553] ldlm_cn02_019 129941 96208106.599123 33184662 120 0.000000 854256.626080 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.790003] ldlm_cn02_027 6493 96208321.624139 33499325 120 0.000000 848452.142837 0.000000 2 / [Fri Apr 12 11:14:25 2019][2937787.801364] [Fri Apr 12 11:14:25 2019][2937787.803038] cpu#31, 1996.203 MHz [Fri Apr 12 11:14:25 2019][2937787.806444] .nr_running : 0 [Fri Apr 12 11:14:25 2019][2937787.811239] .load : 0 [Fri Apr 12 11:14:25 2019][2937787.816040] .nr_switches : 11652282439 [Fri Apr 12 11:14:25 2019][2937787.821698] .nr_load_updates : 1588477665 [Fri Apr 12 11:14:25 2019][2937787.827273] .nr_uninterruptible : 27331 [Fri Apr 12 11:14:25 2019][2937787.832420] .next_balance : 7232.388577 [Fri Apr 12 11:14:25 2019][2937787.838096] .curr->pid : 0 [Fri Apr 12 11:14:25 2019][2937787.842891] .clock : 2937787842.859765 [Fri Apr 12 11:14:25 2019][2937787.849078] .cpu_load[0] : 0 [Fri Apr 12 11:14:25 2019][2937787.853871] .cpu_load[1] : 128 [Fri Apr 12 11:14:25 2019][2937787.858836] .cpu_load[2] : 33 [Fri Apr 12 11:14:25 2019][2937787.863716] .cpu_load[3] : 28 [Fri Apr 12 11:14:25 2019][2937787.868596] .cpu_load[4] : 81 [Fri Apr 12 11:14:26 2019][2937787.873473] .avg_idle : 357509 [Fri Apr 12 11:14:26 2019][2937787.878701] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:26 2019][2937787.884014] [Fri Apr 12 11:14:26 2019][2937787.885685] cfs_rq[31]:/ [Fri Apr 12 11:14:26 2019][2937787.888396] .exec_clock : 0.000000 [Fri Apr 12 11:14:26 2019][2937787.893797] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:26 2019][2937787.899195] .min_vruntime : 64728868.998866 [Fri Apr 12 11:14:26 2019][2937787.905200] .max_vruntime : 0.000001 [Fri Apr 12 11:14:26 2019][2937787.910600] .spread : 0.000000 [Fri Apr 12 11:14:26 2019][2937787.916000] .spread0 : -52115316.856877 [Fri Apr 12 11:14:26 2019][2937787.922093] .nr_spread_over : 0 [Fri Apr 12 11:14:26 2019][2937787.926885] .nr_running : 0 [Fri Apr 12 11:14:26 2019][2937787.931679] .load : 0 [Fri Apr 12 11:14:26 2019][2937787.936515] .runnable_load_avg : 0 [Fri Apr 12 11:14:26 2019][2937787.941306] .blocked_load_avg : 126 [Fri Apr 12 11:14:26 2019][2937787.946281] .tg_load_avg : 0 [Fri Apr 12 11:14:26 2019][2937787.951075] .tg_load_contrib : 0 [Fri Apr 12 11:14:26 2019][2937787.955878] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:26 2019][2937787.960669] .tg->runnable_avg : 0 [Fri Apr 12 11:14:26 2019][2937787.965461] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:26 2019][2937787.970255] .throttled : 0 [Fri Apr 12 11:14:26 2019][2937787.975047] .throttle_count : 0 [Fri Apr 12 11:14:26 2019][2937787.979840] .avg->runnable_avg_sum : 4316 [Fri Apr 12 11:14:26 2019][2937787.984893] .avg->runnable_avg_period : 46818 [Fri Apr 12 11:14:26 2019][2937787.990041] [Fri Apr 12 11:14:26 2019][2937787.991715] rt_rq[31]:/ [Fri Apr 12 11:14:26 2019][2937787.994339] .rt_nr_running : 0 [Fri Apr 12 11:14:26 2019][2937787.999133] .rt_throttled : 0 [Fri Apr 12 11:14:26 2019][2937788.003924] .rt_time : 0.000000 [Fri Apr 12 11:14:26 2019][2937788.009325] .rt_runtime : 950.000000 [Fri Apr 12 11:14:26 2019][2937788.014898] [Fri Apr 12 11:14:26 2019][2937788.016569] runnable tasks: [Fri Apr 12 11:14:26 2019][2937788.019541] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:26 2019][2937788.030488] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:26 2019][2937788.041350] kthreadd 2 64728867.782968 41233 120 0.000000 1253.928505 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.052704] watchdog/31 166 -11.986545 734433 0 0.000000 10120.619131 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.064037] migration/31 167 0.000000 6683 0 0.000000 25301.832339 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.075373] ksoftirqd/31 168 64728752.395426 3969038 120 0.000000 59085.628088 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.086710] kworker/31:0H 170 3930.542998 10 100 0.000000 0.033863 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.098066] kworker/31:1 7306 64728765.061644 1399326 120 0.000000 23722.747036 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.109423] gdbus 18047 64727552.424993 750289 120 0.000000 42927.967688 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.120786] sssd_nss 18105 64728477.751147 1751612 120 0.000000 211951.458025 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.132133] in:imjournal 19305 64728857.212211 3522484 120 0.000000 109037.358294 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.143480] crond 19481 64727552.051234 103722 120 0.000000 16370.125037 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.154816] qmgr 19693 64707711.722518 17994 120 0.000000 1415.442792 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.166155] kworker/31:1H 21319 64728752.377350 2532020 100 0.000000 11583.012369 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.177512] kmmpd-dm-1 117246 64728729.546151 74263 120 0.000000 1134.199379 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.188938] kmmpd-dm-3 117444 64728704.110745 74250 120 0.000000 1216.702150 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.200358] kmmpd-dm-0 117445 64728723.808077 74253 120 0.000000 1137.863155 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.211787] mdt_rdpg03_000 117481 64728735.148583 13689841 120 0.000000 576053.727217 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.223219] mdt_fld_0000 117504 64518061.678030 1372 120 0.000000 24.913609 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.234647] osp-pre-14-2 117581 64728593.115469 41392 120 0.000000 884.752793 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.246074] osp-pre-4-2 117617 64728618.622497 40217 120 0.000000 950.153940 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.257507] osp-syn-36-2 117652 64728669.751461 235438 120 0.000000 6602.585097 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.268932] osp-pre-36-2 117653 64728627.272353 40792 120 0.000000 946.635828 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.280359] lquota_wb_fir-M 117693 64727044.286968 3355 120 0.000000 34.984219 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.291780] osp-pre-12-0 117721 64728588.230325 40148 120 0.000000 857.803595 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.303198] osp-syn-13-0 117722 64678856.363977 74827 120 0.000000 2411.373094 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.314622] osp-pre-13-0 117723 64728704.275543 39289 120 0.000000 1073.934639 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.326053] osp-syn-25-0 117750 64711640.324228 74855 120 0.000000 2179.160044 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.337479] osp-syn-0-0 117754 64711859.871557 75717 120 0.000000 2204.438744 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.348910] osp-syn-1-0 117794 64711717.475710 74848 120 0.000000 2289.761790 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.360332] osp-syn-22-0 117796 64678880.097964 75205 120 0.000000 2315.664280 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.371757] osp-pre-8-0 117807 64728651.313073 39429 120 0.000000 995.789777 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.383180] osp-syn-30-0 117812 64709269.488014 74624 120 0.000000 2284.516352 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.394602] lquota_wb_fir-M 117833 64725679.135477 3319 120 0.000000 38.327867 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.406041] mdt03_014 118230 64728891.002319 15771685 120 0.000000 872552.575802 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.417474] mdt03_028 118348 64728342.678805 8569587 120 0.000000 366470.335723 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.428901] mdt03_033 118381 64383260.302022 461514 120 0.000000 32341.919931 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.440329] mdt03_035 118395 64728342.678744 13392297 120 0.000000 700656.771271 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.451753] mdt03_042 118434 64684775.051624 19994792 120 0.000000 1177528.094121 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.463182] mdt03_053 118545 64728342.678754 13796825 120 0.000000 832600.096178 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.474605] mdt03_055 118550 64383260.302032 15954242 120 0.000000 969403.753100 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.486028] mdt03_056 118551 64195246.157959 4715791 120 0.000000 246708.334193 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.497451] mdt_rdpg03_026 118570 64720781.029444 14583105 120 0.000000 608856.573387 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.508875] mdt03_059 118571 52905642.966096 778069 120 0.000000 26327.374680 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.520297] mdt_rdpg03_027 118581 64728904.796106 16381906 120 0.000000 701641.144635 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.531722] mdt03_069 118622 64728730.331950 16703057 120 0.000000 920555.148122 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.543143] mdt_io03_010 118633 54089485.189557 458480 120 0.000000 17205.915810 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.554574] mdt03_085 118699 64728729.529980 18503299 120 0.000000 1058189.789026 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.566000] mdt03_091 118721 64728342.678273 16128581 120 0.000000 831178.205264 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.577419] mdt03_093 118728 64728342.678504 6660511 120 0.000000 305221.066301 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.588843] mdt03_094 118735 64723791.882557 16312656 120 0.000000 723727.857659 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.600266] mdt03_097 118749 64673698.958740 1224313 120 0.000000 80076.156109 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.611688] mdt03_101 118764 64699833.389832 1241947 120 0.000000 78735.442792 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.623114] mdt03_107 118787 64383260.302162 4008924 120 0.000000 226436.126879 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.634535] mdt03_108 118794 64728729.529068 645887 120 0.000000 41083.683012 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.645957] mdt03_110 118799 64383260.302203 19382613 120 0.000000 1053152.078096 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.657400] mdt_io03_081 118913 61688066.289482 1158298 120 0.000000 82282.288762 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.668867] mdt_rdpg03_039 119217 64728906.186370 15368427 120 0.000000 665562.645516 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.680291] mdt_rdpg03_048 119255 64728902.676922 13381395 120 0.000000 570533.919780 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.691711] mdt_rdpg03_051 119259 64720781.028653 17204829 120 0.000000 674212.076793 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.703154] osp-pre-2-3 2224 64728651.310969 27366 120 0.000000 483.605407 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.714487] osp-pre-1-3 2229 64728661.976434 27334 120 0.000000 476.905465 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.725828] osp-syn-41-3 2278 64728540.851356 9479 120 0.000000 284.930893 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.737158] osp-pre-7-3 2289 64728705.492467 28030 120 0.000000 678.457089 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.748497] osp-pre-0-1 2385 64728631.250754 27369 120 0.000000 568.256332 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.759833] osp-syn-31-1 2406 64725736.181097 20281 120 0.000000 463.716148 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.771168] osp-pre-44-1 2429 64728673.054659 27618 120 0.000000 506.903035 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.782522] osp-syn-21-1 2432 64725920.391995 20301 120 0.000000 482.378177 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.793858] osp-syn-20-1 2444 64725434.973885 20058 120 0.000000 474.833189 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.805192] osp-pre-20-1 2445 64728686.681293 27855 120 0.000000 616.167642 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.816529] osp-pre-45-1 2451 64728704.081480 27696 120 0.000000 587.504532 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.827867] osp-pre-15-1 2461 64728723.795433 27568 120 0.000000 801.090786 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.839203] osp-syn-1-1 2464 64725876.035347 19919 120 0.000000 500.326829 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.850539] osp-syn-27-1 2470 64725708.046955 19932 120 0.000000 517.274847 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.861881] ldlm_cn03_017 18903 64728269.851746 28474562 120 0.000000 577636.552278 0.000000 3 / [Fri Apr 12 11:14:26 2019][2937788.873220] ldlm_cn03_018 18904 64728269.851776 28519732 120 0.000000 581067.471281 0.000000 3 / [Fri Apr 12 11:14:27 2019][2937788.884560] kworker/u260:2 41531 64660901.492905 30901 120 0.000000 4273.612799 0.000000 3 / [Fri Apr 12 11:14:27 2019][2937788.895896] ldlm_cn03_022 44379 64728269.851846 1692404 120 0.000000 48053.805139 0.000000 3 / [Fri Apr 12 11:14:27 2019][2937788.907231] ldlm_cn03_029 45021 64728269.852117 923906 120 0.000000 26937.583510 0.000000 3 / [Fri Apr 12 11:14:27 2019][2937788.918568] kworker/31:0 46406 64674684.739717 2 120 0.000000 0.009447 0.000000 3 / [Fri Apr 12 11:14:27 2019][2937788.929965] [Fri Apr 12 11:14:27 2019][2937788.931637] cpu#32, 1996.203 MHz [Fri Apr 12 11:14:27 2019][2937788.935045] .nr_running : 0 [Fri Apr 12 11:14:27 2019][2937788.939839] .load : 0 [Fri Apr 12 11:14:27 2019][2937788.944639] .nr_switches : 20807324571 [Fri Apr 12 11:14:27 2019][2937788.950299] .nr_load_updates : 1897602958 [Fri Apr 12 11:14:27 2019][2937788.955879] .nr_uninterruptible : -656652 [Fri Apr 12 11:14:27 2019][2937788.961194] .next_balance : 7232.389707 [Fri Apr 12 11:14:27 2019][2937788.966861] .curr->pid : 0 [Fri Apr 12 11:14:27 2019][2937788.971654] .clock : 2937788971.094338 [Fri Apr 12 11:14:27 2019][2937788.977845] .cpu_load[0] : 0 [Fri Apr 12 11:14:27 2019][2937788.982658] .cpu_load[1] : 64 [Fri Apr 12 11:14:27 2019][2937788.987558] .cpu_load[2] : 281 [Fri Apr 12 11:14:27 2019][2937788.992523] .cpu_load[3] : 327 [Fri Apr 12 11:14:27 2019][2937788.997489] .cpu_load[4] : 149 [Fri Apr 12 11:14:27 2019][2937789.002456] .avg_idle : 211353 [Fri Apr 12 11:14:27 2019][2937789.007682] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:27 2019][2937789.012996] [Fri Apr 12 11:14:27 2019][2937789.014672] cfs_rq[32]:/ [Fri Apr 12 11:14:27 2019][2937789.017385] .exec_clock : 0.000000 [Fri Apr 12 11:14:27 2019][2937789.022786] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:27 2019][2937789.028185] .min_vruntime : 115689543.418172 [Fri Apr 12 11:14:27 2019][2937789.034277] .max_vruntime : 0.000001 [Fri Apr 12 11:14:27 2019][2937789.039675] .spread : 0.000000 [Fri Apr 12 11:14:27 2019][2937789.045075] .spread0 : -1154776.652561 [Fri Apr 12 11:14:27 2019][2937789.051080] .nr_spread_over : 0 [Fri Apr 12 11:14:27 2019][2937789.055875] .nr_running : 0 [Fri Apr 12 11:14:27 2019][2937789.060668] .load : 0 [Fri Apr 12 11:14:27 2019][2937789.065462] .runnable_load_avg : 0 [Fri Apr 12 11:14:27 2019][2937789.070253] .blocked_load_avg : 154 [Fri Apr 12 11:14:27 2019][2937789.075218] .tg_load_avg : 0 [Fri Apr 12 11:14:27 2019][2937789.080011] .tg_load_contrib : 0 [Fri Apr 12 11:14:27 2019][2937789.084805] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:27 2019][2937789.089597] .tg->runnable_avg : 0 [Fri Apr 12 11:14:27 2019][2937789.094391] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:27 2019][2937789.099182] .throttled : 0 [Fri Apr 12 11:14:27 2019][2937789.103975] .throttle_count : 0 [Fri Apr 12 11:14:27 2019][2937789.108770] .avg->runnable_avg_sum : 6141 [Fri Apr 12 11:14:27 2019][2937789.113822] .avg->runnable_avg_period : 46207 [Fri Apr 12 11:14:27 2019][2937789.118967] [Fri Apr 12 11:14:27 2019][2937789.120643] rt_rq[32]:/ [Fri Apr 12 11:14:27 2019][2937789.123269] .rt_nr_running : 0 [Fri Apr 12 11:14:27 2019][2937789.128060] .rt_throttled : 0 [Fri Apr 12 11:14:27 2019][2937789.132853] .rt_time : 0.000000 [Fri Apr 12 11:14:27 2019][2937789.138254] .rt_runtime : 950.000000 [Fri Apr 12 11:14:27 2019][2937789.143826] [Fri Apr 12 11:14:27 2019][2937789.145498] runnable tasks: [Fri Apr 12 11:14:27 2019][2937789.148470] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:27 2019][2937789.159417] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:27 2019][2937789.170298] watchdog/32 171 -11.989260 734433 0 0.000000 25953.622096 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.181630] migration/32 172 0.000000 136175 0 0.000000 215064.200240 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.192968] ksoftirqd/32 173 115689529.891608 18305368 120 0.000000 275786.703102 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.204302] kworker/32:0H 175 5239.251168 9 100 0.000000 0.030827 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.215656] scsi_tmf_2 7255 2125.350129 2 100 0.000000 0.009699 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.227009] kworker/32:1H 16838 115689529.881303 15062815 100 0.000000 108955.561522 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.238354] rpciod 17983 12575.737317 2 100 0.000000 0.013656 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.249692] xprtiod 17984 12587.746132 2 100 0.000000 0.009248 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.261051] kworker/32:1 2533 115689560.245916 421145 120 0.000000 12338.479086 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.272391] clio 117068 99060077.838230 2 100 0.000000 0.011502 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.283814] ptlrpc_hr00_002 117075 115689532.456738 2395533 120 0.000000 69880.185911 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.295242] ptlrpcd_00_06 117140 115689500.662451 854372 120 0.000000 23364.285367 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.306670] ldiskfs-rsv-con 117248 99061291.722365 2 100 0.000000 0.008998 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.318093] mgs_params_noti 117281 99061592.068442 2 118 0.000000 0.014056 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.329515] ldiskfs-rsv-con 117449 99062110.342267 2 100 0.000000 0.014598 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.340938] mdt_attr00_000 117483 99062372.985935 2 120 0.000000 0.048262 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.352361] mdt_seqs_0000 117500 99062552.638388 2 120 0.000000 0.053592 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.363782] mdt_seqs_0001 117501 99062564.653881 2 120 0.000000 0.015499 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.375210] osp_up0-2 117546 99064248.316459 2 120 0.000000 0.018475 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.386639] osp_up1-2 117564 99064411.604024 2 120 0.000000 0.028002 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.398077] ldlm_cn00_004 117686 115689502.350262 46385634 120 0.000000 1081057.864753 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.409520] ll_mgs_0005 117903 115579108.079327 630591 120 0.000000 28880.868530 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.420944] mdt_rdpg00_003 118086 115689574.370170 18855217 120 0.000000 986281.789771 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.432369] mdt00_021 118196 115682836.783774 15564674 120 0.000000 929440.311744 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.443798] mdt00_028 118217 115682836.784836 30895174 120 0.000000 1705261.813816 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.455217] mdt00_029 118220 99180501.875996 1000467 120 0.000000 32604.374949 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.466668] mdt00_108 118567 115689529.466378 34526617 120 0.000000 1757938.336279 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.478113] mdt_io00_049 118816 100753799.361018 242591 120 0.000000 11994.554744 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.489546] mdt_io00_057 118857 115661013.930411 2910905 120 0.000000 202081.209387 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.500970] mdt_io00_060 118868 115689423.454197 493182 120 0.000000 36922.007998 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.512417] mdt_io00_100 119035 115689423.448136 2977494 120 0.000000 254408.340415 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.523849] mdt_rdpg00_019 119077 115681848.459340 25587195 120 0.000000 1098206.329289 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.535281] mdt_io00_109 119133 100753799.262590 28 120 0.000000 1.624413 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.546711] mdt_rdpg00_038 119178 115681848.460493 20232174 120 0.000000 996472.242497 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.558143] ldlm_cn00_009 120497 115689580.116396 45498803 120 0.000000 1067382.705150 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.569567] ldlm_cn00_011 121730 115689580.060409 41662252 120 0.000000 1000322.921952 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.580996] ldlm_cn00_015 126538 115689502.414091 35531871 120 0.000000 880721.153039 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.592425] ldlm_cn00_018 127575 115689505.147309 39283352 120 0.000000 953783.442248 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.603844] ldlm_cn00_019 127576 115689507.629815 39255028 120 0.000000 945752.101624 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.615270] ldlm_cn00_023 129947 115689510.798579 34797708 120 0.000000 862833.817302 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.626716] mdt_out00_005 9264 115537199.679954 12936647 120 0.000000 322388.171769 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.638057] kworker/32:2 33497 110586515.673323 5 120 0.000000 0.014787 0.000000 0 / [Fri Apr 12 11:14:27 2019][2937789.649469] [Fri Apr 12 11:14:27 2019][2937789.651143] cpu#33, 1996.203 MHz [Fri Apr 12 11:14:27 2019][2937789.654551] .nr_running : 0 [Fri Apr 12 11:14:27 2019][2937789.659342] .load : 0 [Fri Apr 12 11:14:27 2019][2937789.664136] .nr_switches : 22047086985 [Fri Apr 12 11:14:27 2019][2937789.669795] .nr_load_updates : 1989194399 [Fri Apr 12 11:14:27 2019][2937789.675368] .nr_uninterruptible : 78058 [Fri Apr 12 11:14:27 2019][2937789.680508] .next_balance : 7232.390428 [Fri Apr 12 11:14:27 2019][2937789.686167] .curr->pid : 0 [Fri Apr 12 11:14:27 2019][2937789.690959] .clock : 2937789690.881483 [Fri Apr 12 11:14:27 2019][2937789.697152] .cpu_load[0] : 0 [Fri Apr 12 11:14:27 2019][2937789.701950] .cpu_load[1] : 641 [Fri Apr 12 11:14:27 2019][2937789.706914] .cpu_load[2] : 99 [Fri Apr 12 11:14:27 2019][2937789.711795] .cpu_load[3] : 285 [Fri Apr 12 11:14:27 2019][2937789.716759] .cpu_load[4] : 290 [Fri Apr 12 11:14:27 2019][2937789.721729] .avg_idle : 86179 [Fri Apr 12 11:14:27 2019][2937789.726868] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:27 2019][2937789.732181] [Fri Apr 12 11:14:27 2019][2937789.733857] cfs_rq[33]:/ [Fri Apr 12 11:14:27 2019][2937789.736569] .exec_clock : 0.000000 [Fri Apr 12 11:14:27 2019][2937789.741969] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:27 2019][2937789.747369] .min_vruntime : 130360158.993938 [Fri Apr 12 11:14:27 2019][2937789.753462] .max_vruntime : 0.000001 [Fri Apr 12 11:14:27 2019][2937789.758863] .spread : 0.000000 [Fri Apr 12 11:14:27 2019][2937789.764262] .spread0 : 13515784.082960 [Fri Apr 12 11:14:27 2019][2937789.770266] .nr_spread_over : 0 [Fri Apr 12 11:14:27 2019][2937789.775060] .nr_running : 0 [Fri Apr 12 11:14:27 2019][2937789.779854] .load : 2048 [Fri Apr 12 11:14:27 2019][2937789.784905] .runnable_load_avg : 0 [Fri Apr 12 11:14:27 2019][2937789.789699] .blocked_load_avg : 364 [Fri Apr 12 11:14:27 2019][2937789.794664] .tg_load_avg : 0 [Fri Apr 12 11:14:27 2019][2937789.799457] .tg_load_contrib : 0 [Fri Apr 12 11:14:27 2019][2937789.804250] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:27 2019][2937789.809041] .tg->runnable_avg : 0 [Fri Apr 12 11:14:27 2019][2937789.813835] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:27 2019][2937789.818627] .throttled : 0 [Fri Apr 12 11:14:27 2019][2937789.823421] .throttle_count : 0 [Fri Apr 12 11:14:27 2019][2937789.828215] .avg->runnable_avg_sum : 15793 [Fri Apr 12 11:14:27 2019][2937789.833353] .avg->runnable_avg_period : 46991 [Fri Apr 12 11:14:27 2019][2937789.838500] [Fri Apr 12 11:14:27 2019][2937789.840173] rt_rq[33]:/ [Fri Apr 12 11:14:27 2019][2937789.842799] .rt_nr_running : 0 [Fri Apr 12 11:14:27 2019][2937789.847591] .rt_throttled : 0 [Fri Apr 12 11:14:27 2019][2937789.852386] .rt_time : 0.000000 [Fri Apr 12 11:14:27 2019][2937789.857785] .rt_runtime : 950.000000 [Fri Apr 12 11:14:28 2019][2937789.863357] [Fri Apr 12 11:14:28 2019][2937789.865031] runnable tasks: [Fri Apr 12 11:14:28 2019][2937789.868003] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:28 2019][2937789.878948] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:28 2019][2937789.889831] watchdog/33 176 -11.986495 734433 0 0.000000 37022.702864 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.901170] migration/33 177 0.000000 146538 0 0.000000 337717.018154 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.912506] ksoftirqd/33 178 130358124.594853 12767083 120 0.000000 304825.636685 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.923845] kworker/33:0H 180 18303.217080 10 100 0.000000 0.036911 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.935189] netns 256 47.347260 2 100 0.000000 0.009819 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.946524] kintegrityd 259 83.373204 2 100 0.000000 0.009047 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.957861] bioset 260 95.381780 2 100 0.000000 0.009167 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.969198] bioset 261 107.390146 2 100 0.000000 0.008947 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.980532] bioset 262 119.398502 2 100 0.000000 0.008937 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937789.991870] kblockd 263 131.406357 2 100 0.000000 0.008427 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.003207] md 264 143.414442 2 100 0.000000 0.008766 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.014542] edac-poller 265 155.422699 2 100 0.000000 0.008937 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.025877] watchdogd 266 167.430995 2 100 0.000000 0.008887 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.037216] irq/27-AMD-Vi 270 0.000000 2 49 0.000000 0.018145 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.048553] kswapd1 278 492.023919 3 120 0.000000 0.022172 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.059887] ksmd 281 347.689061 2 125 0.000000 0.010270 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.071224] crypto 283 371.788564 2 100 0.000000 0.009508 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.082560] kthrotld 291 467.990373 2 100 0.000000 0.009737 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.093897] kmpath_rdacd 296 528.044884 2 100 0.000000 0.018084 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.105232] kaluad 297 540.060084 2 100 0.000000 0.015820 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.116568] kpsmoused 301 564.088701 2 100 0.000000 0.019538 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.127904] ipv6_addrconf 303 576.106967 2 100 0.000000 0.018957 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.139241] deferwq 317 648.262366 2 100 0.000000 0.013946 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.150578] scsi_eh_0 1956 1542.586044 2 120 0.000000 0.020008 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.161921] ccp-1-q3 12389 4180.342790 2 120 0.000000 0.011221 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.173277] gssproxy 18071 12167.202666 1 120 0.000000 0.016351 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.184611] gssproxy 18073 12179.209523 1 120 0.000000 0.006863 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.195958] kworker/33:1H 20100 130358124.585790 8383356 100 0.000000 66713.694973 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.207298] kworker/33:2 62438 112892519.579219 1110034 120 0.000000 11133.647520 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.218637] ptlrpc_hr01_001 117081 130360214.826789 3117528 120 0.000000 110232.820124 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.230065] kiblnd_sd_01_01 117120 130360260.998759 2224216682 120 0.000000 21826035.795861 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.241579] ldlm_cn01_001 117270 130359755.441010 53228636 120 0.000000 1302840.236351 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.253004] ldiskfs-rsv-con 117448 112644822.829630 2 100 0.000000 0.010890 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.264438] osp_up3-2 117566 112644994.784326 2 120 0.000000 0.028865 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.275874] ldlm_cn01_003 117681 130360246.530182 56777646 120 0.000000 1373704.996594 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.287320] mdt_rdpg01_003 117908 130349351.027324 31589099 120 0.000000 1536465.711161 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.298748] mdt_rdpg01_005 118026 130352969.632834 26986149 120 0.000000 1350677.110404 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.310171] mdt01_013 118155 120852900.041193 7637063 120 0.000000 484382.367000 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.321606] mdt01_058 118309 121143693.206542 14378062 120 0.000000 708878.374322 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.333032] mdt01_064 118323 130360267.156487 21816631 120 0.000000 1610675.382636 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.344457] mdt01_080 118344 124434650.864528 17470352 120 0.000000 1022246.458785 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.355881] mdt01_092 118370 120551655.423256 14497015 120 0.000000 665960.988701 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.367301] mdt01_100 118393 130297010.384223 21511889 120 0.000000 1543507.374254 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.378730] mdt01_111 118458 130297838.267657 20472388 120 0.000000 1470104.556161 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.390166] mdt_io01_008 118600 129952029.140910 835865 120 0.000000 34367.904406 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.401601] mdt_io01_021 118731 129639705.535413 1887144 120 0.000000 167497.039685 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.413030] mdt_io01_040 118845 129825709.083332 1203759 120 0.000000 100308.610987 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.424457] mdt_io01_048 118887 120153049.178273 381412 120 0.000000 32825.280628 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.435893] mdt_io01_061 118972 120153049.309492 1511761 120 0.000000 80137.758644 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.447317] mdt_io01_069 119015 120153049.166459 73563 120 0.000000 3393.478357 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.458737] mdt_io01_070 119020 129639705.535323 1801908 120 0.000000 134931.948322 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.470164] mdt_io01_075 119049 129639705.535544 2742972 120 0.000000 226186.171327 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.481586] mdt_rdpg01_016 119068 130352970.067007 27811369 120 0.000000 1295677.292043 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.493024] mdt_rdpg01_019 119083 130352969.635087 29696066 120 0.000000 1324261.294651 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.504448] mdt_io01_086 119105 120153049.091989 397893 120 0.000000 21808.055899 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.515869] mdt_rdpg01_029 119108 129034917.422779 26689667 120 0.000000 1392631.520345 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.527292] mdt_io01_087 119111 120153332.439767 31014 120 0.000000 1397.971250 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.538719] mdt_io01_097 119146 120153049.412647 489980 120 0.000000 17208.861449 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.550143] mdt_io01_101 119175 129639705.535413 878250 120 0.000000 98280.233083 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.561568] mdt_rdpg01_044 119187 130352970.060725 28296443 120 0.000000 1299564.162155 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.572993] mdt_rdpg01_052 119213 130352970.060624 17242247 120 0.000000 960971.515602 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.584415] mdt_io01_110 119223 129639705.535473 2594739 120 0.000000 193176.441584 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.595850] kworker/33:0 121822 130360249.904679 267423 120 0.000000 11987.180398 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.607270] ldlm_cn01_012 122661 130359755.440951 49467362 120 0.000000 1259096.933834 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.618693] R ldlm_cn01_014 123223 123976643.824950 47210868 120 0.000000 1218277.145938 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.630137] lquota_wb_fir-M 2348 130355196.871331 4426 120 0.000000 144.016120 0.000000 1 / [Fri Apr 12 11:14:28 2019][2937790.641593] [Fri Apr 12 11:14:28 2019][2937790.643277] cpu#34, 1996.203 MHz [Fri Apr 12 11:14:28 2019][2937790.646692] .nr_running : 0 [Fri Apr 12 11:14:28 2019][2937790.651493] .load : 1024 [Fri Apr 12 11:14:28 2019][2937790.656554] .nr_switches : 9976589587 [Fri Apr 12 11:14:28 2019][2937790.662135] .nr_load_updates : 1712083278 [Fri Apr 12 11:14:28 2019][2937790.667706] .nr_uninterruptible : 96044 [Fri Apr 12 11:14:28 2019][2937790.672847] .next_balance : 7232.391419 [Fri Apr 12 11:14:28 2019][2937790.678505] .curr->pid : 0 [Fri Apr 12 11:14:28 2019][2937790.683299] .clock : 2937790683.282977 [Fri Apr 12 11:14:28 2019][2937790.689480] .cpu_load[0] : 1024 [Fri Apr 12 11:14:28 2019][2937790.694540] .cpu_load[1] : 216 [Fri Apr 12 11:14:28 2019][2937790.699517] .cpu_load[2] : 88 [Fri Apr 12 11:14:28 2019][2937790.704402] .cpu_load[3] : 171 [Fri Apr 12 11:14:28 2019][2937790.709369] .cpu_load[4] : 150 [Fri Apr 12 11:14:28 2019][2937790.714363] .avg_idle : 232574 [Fri Apr 12 11:14:28 2019][2937790.719596] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:28 2019][2937790.724909] [Fri Apr 12 11:14:28 2019][2937790.726586] cfs_rq[34]:/ [Fri Apr 12 11:14:28 2019][2937790.729298] .exec_clock : 0.000000 [Fri Apr 12 11:14:28 2019][2937790.734699] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:28 2019][2937790.740099] .min_vruntime : 77853996.619809 [Fri Apr 12 11:14:28 2019][2937790.746103] .max_vruntime : 0.000001 [Fri Apr 12 11:14:28 2019][2937790.751503] .spread : 0.000000 [Fri Apr 12 11:14:28 2019][2937790.756903] .spread0 : -38990547.835300 [Fri Apr 12 11:14:28 2019][2937790.762996] .nr_spread_over : 0 [Fri Apr 12 11:14:28 2019][2937790.767788] .nr_running : 0 [Fri Apr 12 11:14:28 2019][2937790.772582] .load : 0 [Fri Apr 12 11:14:28 2019][2937790.777373] .runnable_load_avg : 0 [Fri Apr 12 11:14:28 2019][2937790.782165] .blocked_load_avg : 262 [Fri Apr 12 11:14:28 2019][2937790.787133] .tg_load_avg : 0 [Fri Apr 12 11:14:28 2019][2937790.791925] .tg_load_contrib : 0 [Fri Apr 12 11:14:28 2019][2937790.796718] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:28 2019][2937790.801511] .tg->runnable_avg : 0 [Fri Apr 12 11:14:28 2019][2937790.806305] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:28 2019][2937790.811106] .throttled : 0 [Fri Apr 12 11:14:28 2019][2937790.815898] .throttle_count : 0 [Fri Apr 12 11:14:28 2019][2937790.820692] .avg->runnable_avg_sum : 9660 [Fri Apr 12 11:14:28 2019][2937790.825743] .avg->runnable_avg_period : 46319 [Fri Apr 12 11:14:28 2019][2937790.830894] [Fri Apr 12 11:14:28 2019][2937790.832565] rt_rq[34]:/ [Fri Apr 12 11:14:28 2019][2937790.835191] .rt_nr_running : 0 [Fri Apr 12 11:14:28 2019][2937790.839984] .rt_throttled : 0 [Fri Apr 12 11:14:28 2019][2937790.844777] .rt_time : 0.000000 [Fri Apr 12 11:14:28 2019][2937790.850177] .rt_runtime : 950.000000 [Fri Apr 12 11:14:28 2019][2937790.855749] [Fri Apr 12 11:14:28 2019][2937790.857421] runnable tasks: [Fri Apr 12 11:14:28 2019][2937790.860394] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:29 2019][2937790.871340] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:29 2019][2937790.882224] watchdog/34 181 -11.986585 734433 0 0.000000 10187.218173 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.893562] migration/34 182 0.000000 89023 0 0.000000 42857.524694 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.904898] ksoftirqd/34 183 77853978.156000 17092277 120 0.000000 618502.740693 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.916234] kworker/34:0H 185 5601.690342 9 100 0.000000 0.029484 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.927603] kworker/34:1H 16839 77853837.897298 6292499 100 0.000000 36282.480476 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.938955] dsm_sa_datamgrd 19149 14981.587692 1 120 0.000000 0.049995 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.950286] dsm_sa_datamgrd 19153 15005.609481 1 120 0.000000 0.009498 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.961626] writer#3 19294 77853283.950796 15207807 120 0.000000 236145.418659 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.972961] nfsiod 19365 15546.762869 2 100 0.000000 0.010861 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.984298] agetty 19552 16184.050929 9 120 0.000000 10.647935 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937790.995670] kworker/34:0 116936 77853944.605671 279195 120 0.000000 15059.538985 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.007101] ptlrpcd_02_11 117169 77853836.985510 860532 120 0.000000 21608.600408 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.018526] ll_mgs_0001 117283 77853328.380965 4416272 120 0.000000 139806.411481 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.029955] mdt_io02_002 117515 64520658.372461 205 120 0.000000 10.980378 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.041378] mdt02_007 117536 77853522.287007 18250802 120 0.000000 1186067.781969 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.052822] dist_txn-2 117697 62904084.438923 2 120 0.000000 0.029716 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.064277] mdt02_011 118171 76696883.321669 21707039 120 0.000000 1291133.389065 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.075707] mdt02_027 118252 63336518.244672 51661 120 0.000000 2421.365722 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.087127] mdt02_028 118258 63466936.265042 3627213 120 0.000000 144106.185783 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.098560] mdt02_044 118364 77733100.937245 15496944 120 0.000000 1059024.526203 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.109999] mdt02_057 118442 77853522.294220 22545104 120 0.000000 1450641.727726 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.121423] mdt02_059 118450 77853698.968565 24126346 120 0.000000 1415352.349039 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.132849] mdt02_073 118487 77840309.015208 18845836 120 0.000000 1265637.576157 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.144279] R mdt02_077 118504 77854071.327233 13005095 120 0.000000 917235.976266 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.155716] mdt_io02_019 118651 64520376.956471 79 120 0.000000 4.310722 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.167147] mdt_io02_038 118713 64520659.337547 193 120 0.000000 9.598361 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.178577] mdt_io02_057 118791 64520378.690242 46 120 0.000000 2.027825 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.190005] mdt_io02_059 118801 64520377.710310 63 120 0.000000 2.622986 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.201441] mdt_io02_076 118877 64520365.005641 42 120 0.000000 2.804082 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.212869] mdt_io02_081 118894 64520404.078493 567422 120 0.000000 25425.908272 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.224310] mdt_io02_087 118911 64520377.617273 49 120 0.000000 2.701592 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.235735] mdt_io02_096 118944 64520658.104367 66 120 0.000000 2.968259 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.247198] mdt_rdpg02_015 119062 77854020.505120 22929047 120 0.000000 1145650.492046 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.258920] mdt_rdpg02_024 119091 77852145.509721 19596732 120 0.000000 971770.489667 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.270374] mdt_rdpg02_037 119159 77811287.278406 16984970 120 0.000000 940674.624708 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.281800] mdt_rdpg02_048 119195 77854090.184783 14116735 120 0.000000 740455.109331 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.293282] kworker/34:2 29749 71350571.126152 2 120 0.000000 0.008696 0.000000 2 / [Fri Apr 12 11:14:29 2019][2937791.304681] [Fri Apr 12 11:14:29 2019][2937791.306360] cpu#35, 1996.203 MHz [Fri Apr 12 11:14:29 2019][2937791.309765] .nr_running : 1 [Fri Apr 12 11:14:29 2019][2937791.314570] .load : 0 [Fri Apr 12 11:14:29 2019][2937791.319369] .nr_switches : 9459989779 [Fri Apr 12 11:14:29 2019][2937791.324941] .nr_load_updates : 1537243849 [Fri Apr 12 11:14:29 2019][2937791.330513] .nr_uninterruptible : 32853 [Fri Apr 12 11:14:29 2019][2937791.335654] .next_balance : 7232.392083 [Fri Apr 12 11:14:29 2019][2937791.341314] .curr->pid : 0 [Fri Apr 12 11:14:29 2019][2937791.346107] .clock : 2937791345.988463 [Fri Apr 12 11:14:29 2019][2937791.352304] .cpu_load[0] : 0 [Fri Apr 12 11:14:29 2019][2937791.357105] .cpu_load[1] : 0 [Fri Apr 12 11:14:29 2019][2937791.361907] .cpu_load[2] : 2 [Fri Apr 12 11:14:29 2019][2937791.366718] .cpu_load[3] : 12 [Fri Apr 12 11:14:29 2019][2937791.371607] .cpu_load[4] : 33 [Fri Apr 12 11:14:29 2019][2937791.376494] .avg_idle : 465366 [Fri Apr 12 11:14:29 2019][2937791.381727] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:29 2019][2937791.387052] [Fri Apr 12 11:14:29 2019][2937791.388727] cfs_rq[35]:/ [Fri Apr 12 11:14:29 2019][2937791.391440] .exec_clock : 0.000000 [Fri Apr 12 11:14:29 2019][2937791.396839] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:29 2019][2937791.402237] .min_vruntime : 66803432.518289 [Fri Apr 12 11:14:29 2019][2937791.408244] .max_vruntime : 0.000001 [Fri Apr 12 11:14:29 2019][2937791.413643] .spread : 0.000000 [Fri Apr 12 11:14:29 2019][2937791.419043] .spread0 : -50041164.028312 [Fri Apr 12 11:14:29 2019][2937791.425136] .nr_spread_over : 0 [Fri Apr 12 11:14:29 2019][2937791.429928] .nr_running : 1 [Fri Apr 12 11:14:29 2019][2937791.434722] .load : 0 [Fri Apr 12 11:14:29 2019][2937791.439515] .runnable_load_avg : 0 [Fri Apr 12 11:14:29 2019][2937791.444308] .blocked_load_avg : 58 [Fri Apr 12 11:14:29 2019][2937791.449188] .tg_load_avg : 0 [Fri Apr 12 11:14:29 2019][2937791.453979] .tg_load_contrib : 0 [Fri Apr 12 11:14:29 2019][2937791.458781] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:29 2019][2937791.463574] .tg->runnable_avg : 0 [Fri Apr 12 11:14:29 2019][2937791.468366] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:29 2019][2937791.473159] .throttled : 0 [Fri Apr 12 11:14:29 2019][2937791.477951] .throttle_count : 0 [Fri Apr 12 11:14:29 2019][2937791.482745] .avg->runnable_avg_sum : 5297 [Fri Apr 12 11:14:29 2019][2937791.487797] .avg->runnable_avg_period : 46204 [Fri Apr 12 11:14:29 2019][2937791.492945] [Fri Apr 12 11:14:29 2019][2937791.494619] rt_rq[35]:/ [Fri Apr 12 11:14:29 2019][2937791.497243] .rt_nr_running : 0 [Fri Apr 12 11:14:29 2019][2937791.502038] .rt_throttled : 0 [Fri Apr 12 11:14:29 2019][2937791.506830] .rt_time : 0.007815 [Fri Apr 12 11:14:29 2019][2937791.512230] .rt_runtime : 950.000000 [Fri Apr 12 11:14:29 2019][2937791.517803] [Fri Apr 12 11:14:29 2019][2937791.519473] runnable tasks: [Fri Apr 12 11:14:29 2019][2937791.522446] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:29 2019][2937791.533392] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:29 2019][2937791.544276] watchdog/35 186 -11.985393 734434 0 0.000000 10730.819570 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.555614] migration/35 187 0.000000 108195 0 0.000000 42716.310197 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.566951] ksoftirqd/35 188 66803421.290481 4979908 120 0.000000 91446.379712 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.578286] kworker/35:0H 190 9445.936195 9 100 0.000000 0.032522 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.589638] mlx5_health0000 7283 2689.000589 2 100 0.000000 0.011491 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.600978] mlx5_cmd_0000:8 7285 2713.075877 2 100 0.000000 0.009648 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.612314] mlx5_page_alloc 7348 3119.735131 2 100 0.000000 0.009739 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.623650] kworker/u265:0 7398 3304.704944 4 100 0.000000 0.014647 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.634987] ib-comp-wq 7412 3316.704905 2 100 0.000000 0.010410 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.646322] ib-comp-unb-wq 7413 3328.714185 2 100 0.000000 0.010100 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.657659] ib_mcast 7414 3340.722963 2 100 0.000000 0.009438 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.669003] ib_nl_sa_wq 7415 3352.732011 2 100 0.000000 0.009698 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.680339] mlx5_ib_sigerr_ 7423 3400.933028 2 100 0.000000 0.010490 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.691684] ib_mad1 7428 3461.195743 2 100 0.000000 0.010720 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.703020] to_fifo 7429 3473.204931 2 100 0.000000 0.009838 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.714365] mkey_cache 7432 3509.340831 2 100 0.000000 0.011002 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.725705] poll_mpt3sas0_s 7451 3617.659764 2 100 0.000000 0.010761 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.737046] ccp-6-q4 13603 6255.555886 2 120 0.000000 0.012544 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.748389] kworker/35:1H 16986 66803438.695135 3025635 100 0.000000 14665.864586 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.759733] irqbalance 18015 66803182.016403 492924 120 0.000000 2377165.291085 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.771104] gssproxy 18066 15133.644920 2 120 0.000000 3.601067 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.782438] sssd_pam 18106 66803301.017187 587270 120 0.000000 12574.000365 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.793778] nfsv4.1-svc 19435 18324.668645 2 120 0.000000 0.012203 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.805131] kworker/35:2 104646 66803412.058937 389036 120 0.000000 7549.273063 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.816561] cfs_rh_03 117055 66801470.178960 9905 120 0.000000 6194.186619 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.828023] ll_mgs_0003 117297 66311389.683244 1670526 120 0.000000 45500.183581 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.839452] mdt_out03_001 117499 66803374.631279 1211 120 0.000000 80.049157 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.850885] osp-pre-0-2 117547 66803374.642981 37051 120 0.000000 811.980347 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.862316] osp-pre-9-2 117589 66803414.087613 41432 120 0.000000 1072.065973 0.000000 3 / [Fri Apr 12 11:14:29 2019][2937791.873744] osp-pre-32-2 117609 66803419.416153 40367 120 0.000000 998.208917 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.885256] osp-pre-24-2 117633 66803345.747746 41762 120 0.000000 1052.462367 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.896691] osp-pre-15-2 117639 66803300.990106 39936 120 0.000000 934.539428 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.908112] osp-pre-1-2 117643 66803106.268140 39962 120 0.000000 963.085372 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.919547] osp-pre-11-0 117731 66803319.551395 40178 120 0.000000 889.933387 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.930982] osp-syn-29-0 117792 66789237.982369 75412 120 0.000000 2326.850195 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.942408] osp-syn-3-0 117798 66313415.548798 74977 120 0.000000 2265.922835 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.953848] osp-syn-27-0 117800 66787849.810876 75844 120 0.000000 2253.816094 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.965273] osp-pre-8-0 117807 66803304.990041 39430 120 0.000000 995.805277 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.976702] mdt_rdpg03_003 118083 66798358.293312 20436266 120 0.000000 870805.567746 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.988166] mdt_rdpg03_012 118509 66785516.838670 18904101 120 0.000000 786827.098864 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937791.999593] mdt_rdpg03_020 118535 66794670.504470 16855354 120 0.000000 650741.507817 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.011016] mdt_rdpg03_023 118542 66798358.293966 14036663 120 0.000000 631966.058298 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.022442] mdt03_063 118589 66317671.871881 22775676 120 0.000000 1194152.351430 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.033871] mdt_io03_015 118659 66803458.877339 1268465 120 0.000000 91821.973552 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.045293] mdt03_076 118662 66796805.931305 8105220 120 0.000000 325024.383963 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.056721] mdt_io03_027 118717 66627664.931013 2224225 120 0.000000 130698.537359 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.068147] mdt_io03_029 118723 55834435.815635 37 120 0.000000 1.518724 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.079573] mdt_io03_035 118745 66788681.880283 1786712 120 0.000000 106417.594063 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.091073] mdt_rdpg03_055 119263 66794670.503669 13428102 120 0.000000 561238.684658 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.102495] mdt_rdpg03_057 119265 66798358.293074 15600556 120 0.000000 628338.079599 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.113924] ldlm_cn03_010 122828 66802388.493113 46119264 120 0.000000 887550.180223 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.125365] osp-pre-46-3 2239 66803349.077410 28005 120 0.000000 556.283638 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.136706] osp-syn-9-3 2252 66801665.244963 9549 120 0.000000 302.785905 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.148040] osp-pre-32-3 2273 66803173.332055 28017 120 0.000000 506.173608 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.159374] osp-pre-16-3 2277 66803402.549027 28024 120 0.000000 511.585552 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.170711] osp-pre-40-3 2291 66803201.333545 28032 120 0.000000 654.914560 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.182066] osp-pre-29-3 2305 66803355.767878 28006 120 0.000000 638.845681 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.193400] osp-syn-1-3 2306 66801936.331452 9516 120 0.000000 287.492569 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.204737] osp-syn-22-3 2308 66802332.886729 9383 120 0.000000 284.900932 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.216091] osp-syn-27-3 2312 66803402.568604 9552 120 0.000000 279.946763 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.227426] osp-pre-37-3 2315 66803438.159706 28027 120 0.000000 516.986672 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.238783] osp-pre-0-1 2385 66803286.643788 27370 120 0.000000 568.269407 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.250116] osp-syn-46-1 2396 66799160.466816 19956 120 0.000000 461.684598 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.261455] osp-syn-14-1 2402 66799086.149875 20134 120 0.000000 482.112651 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.272789] osp-pre-14-1 2403 66803412.454355 27861 120 0.000000 604.142075 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.284125] osp-syn-43-1 2414 66799294.681958 20104 120 0.000000 462.583679 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.295462] osp-pre-23-1 2441 66803409.011394 27568 120 0.000000 543.117823 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.306798] osp-pre-7-1 2447 66803411.231979 27847 120 0.000000 642.752003 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.318135] osp-pre-38-1 2459 66803172.514714 27587 120 0.000000 648.462498 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.329470] osp-pre-1-1 2465 66803300.794797 28054 120 0.000000 671.746701 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.340808] osp-syn-3-1 2468 66799088.596268 19620 120 0.000000 508.898716 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.352145] mdt_io03_088 3058 55834435.867906 15 120 0.000000 0.692687 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.363479] mdt_io03_094 3065 55834436.313753 10 120 0.000000 0.379570 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.374816] mdt_io03_097 3068 66627664.929680 2208000 120 0.000000 127055.515228 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.386161] kworker/35:0 38160 63150902.218405 2 120 0.000000 0.009659 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.397500] ldlm_cn03_031 45023 66802388.492943 1060925 120 0.000000 30819.726771 0.000000 3 / [Fri Apr 12 11:14:30 2019][2937792.408895] [Fri Apr 12 11:14:30 2019][2937792.410571] cpu#36, 1996.203 MHz [Fri Apr 12 11:14:30 2019][2937792.413977] .nr_running : 1 [Fri Apr 12 11:14:30 2019][2937792.418771] .load : 0 [Fri Apr 12 11:14:30 2019][2937792.423571] .nr_switches : 14960231592 [Fri Apr 12 11:14:30 2019][2937792.429230] .nr_load_updates : 1835990325 [Fri Apr 12 11:14:30 2019][2937792.434806] .nr_uninterruptible : -830923 [Fri Apr 12 11:14:30 2019][2937792.440128] .next_balance : 7232.393185 [Fri Apr 12 11:14:30 2019][2937792.445805] .curr->pid : 0 [Fri Apr 12 11:14:30 2019][2937792.450604] .clock : 2937792450.442038 [Fri Apr 12 11:14:30 2019][2937792.456793] .cpu_load[0] : 0 [Fri Apr 12 11:14:30 2019][2937792.461594] .cpu_load[1] : 74 [Fri Apr 12 11:14:30 2019][2937792.466481] .cpu_load[2] : 308 [Fri Apr 12 11:14:30 2019][2937792.471450] .cpu_load[3] : 657 [Fri Apr 12 11:14:30 2019][2937792.476422] .cpu_load[4] : 540 [Fri Apr 12 11:14:30 2019][2937792.481406] .avg_idle : 65397 [Fri Apr 12 11:14:30 2019][2937792.486554] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:30 2019][2937792.491874] [Fri Apr 12 11:14:30 2019][2937792.493545] cfs_rq[36]:/ [Fri Apr 12 11:14:30 2019][2937792.496265] .exec_clock : 0.000000 [Fri Apr 12 11:14:30 2019][2937792.501666] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:30 2019][2937792.507065] .min_vruntime : 103979787.950637 [Fri Apr 12 11:14:30 2019][2937792.513156] .max_vruntime : 0.000001 [Fri Apr 12 11:14:30 2019][2937792.518555] .spread : 0.000000 [Fri Apr 12 11:14:30 2019][2937792.523956] .spread0 : -12864975.404432 [Fri Apr 12 11:14:30 2019][2937792.530048] .nr_spread_over : 0 [Fri Apr 12 11:14:30 2019][2937792.534841] .nr_running : 0 [Fri Apr 12 11:14:30 2019][2937792.539634] .load : 0 [Fri Apr 12 11:14:30 2019][2937792.544427] .runnable_load_avg : 0 [Fri Apr 12 11:14:30 2019][2937792.549221] .blocked_load_avg : 297 [Fri Apr 12 11:14:30 2019][2937792.554185] .tg_load_avg : 0 [Fri Apr 12 11:14:30 2019][2937792.558978] .tg_load_contrib : 0 [Fri Apr 12 11:14:30 2019][2937792.563772] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:30 2019][2937792.568565] .tg->runnable_avg : 0 [Fri Apr 12 11:14:30 2019][2937792.573358] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:30 2019][2937792.578151] .throttled : 0 [Fri Apr 12 11:14:30 2019][2937792.582943] .throttle_count : 0 [Fri Apr 12 11:14:30 2019][2937792.587737] .avg->runnable_avg_sum : 10043 [Fri Apr 12 11:14:30 2019][2937792.592874] .avg->runnable_avg_period : 46359 [Fri Apr 12 11:14:30 2019][2937792.598027] [Fri Apr 12 11:14:30 2019][2937792.599704] rt_rq[36]:/ [Fri Apr 12 11:14:30 2019][2937792.602329] .rt_nr_running : 0 [Fri Apr 12 11:14:30 2019][2937792.607123] .rt_throttled : 0 [Fri Apr 12 11:14:30 2019][2937792.611916] .rt_time : 0.000000 [Fri Apr 12 11:14:30 2019][2937792.617316] .rt_runtime : 950.000000 [Fri Apr 12 11:14:30 2019][2937792.622887] [Fri Apr 12 11:14:30 2019][2937792.624570] runnable tasks: [Fri Apr 12 11:14:30 2019][2937792.627541] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:30 2019][2937792.638487] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:30 2019][2937792.649371] watchdog/36 191 -11.987907 734435 0 0.000000 25166.957445 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.660719] migration/36 192 0.000000 130585 0 0.000000 270531.187258 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.672056] ksoftirqd/36 193 103979801.334007 21906833 120 0.000000 321461.335322 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.683399] kworker/36:0H 195 12563.538211 9 100 0.000000 0.026689 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.694755] ccp-1-q2 12388 4434.609476 2 120 0.000000 0.009418 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.706090] ccp-2-q1 13499 4609.560514 2 120 0.000000 0.008947 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.717440] kworker/36:1H 17647 103979806.049777 18438527 100 0.000000 138214.837884 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.728790] dsm_sa_eventmgr 18032 13574.205718 5 120 0.000000 2.172072 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.740157] kiblnd_sd_00_03 117118 103979823.650379 2268891609 120 0.000000 19924190.862998 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.751677] ldlm_cb00_000 117256 103894710.371101 789 120 0.000000 24.607056 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.763180] mdt00_043 118260 89259833.356272 9242644 120 0.000000 343586.546541 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.774603] mdt00_048 118278 103979732.767024 34487625 120 0.000000 1916953.269830 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.786024] mdt00_051 118286 103972157.672513 22581457 120 0.000000 1218621.950908 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.797515] mdt00_063 118363 103979492.373412 27107041 120 0.000000 1389018.026762 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.808992] mdt00_067 118386 103979820.973707 32432448 120 0.000000 1862192.410905 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.820515] mdt_io00_073 118935 103935918.006665 1374227 120 0.000000 96169.782415 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.831942] mdt_io00_075 118945 103935918.006614 77 120 0.000000 5.103446 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.843404] mdt_io00_082 118966 89561866.725743 92 120 0.000000 4.785615 0.000000 0 / [Fri Apr 12 11:14:30 2019][2937792.854890] mdt_io00_093 119014 89561379.693297 102 120 0.000000 3.211557 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.866328] mdt_rdpg00_014 119040 103972146.840566 21969695 120 0.000000 1006913.148200 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.877763] mdt_rdpg00_016 119043 103972146.836238 17914189 120 0.000000 931871.618930 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.889225] mdt_rdpg00_027 119126 103979193.573330 20597931 120 0.000000 1007558.178767 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.900673] mdt_rdpg00_033 119154 103972645.417177 24752966 120 0.000000 1058114.776786 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.912138] ldiskfs-rsv-con 2219 89557829.320618 2 100 0.000000 0.011442 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.923498] osp_up2-3 2223 89557860.088641 2 120 0.000000 0.037181 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.934835] osp_up0-3 2226 89557872.110877 2 120 0.000000 0.022242 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.946188] osp_up1-3 2228 89557896.175286 2 120 0.000000 0.021139 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.957551] osp-pre-28-3 2323 103979063.590706 27862 120 0.000000 600.614555 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.968948] dist_txn-3 2349 89558483.235965 2 120 0.000000 0.025058 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.980301] ldiskfs-rsv-con 2370 89558700.899139 2 100 0.000000 0.013525 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937792.991656] kworker/36:1 30971 103979836.491947 81130 120 0.000000 5324.567339 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937793.003081] kworker/36:2 32026 103406774.103026 4 120 0.000000 0.147671 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937793.014895] kworker/u257:1 40885 103946401.500774 8045 120 0.000000 1227.398738 0.000000 0 / [Fri Apr 12 11:14:31 2019][2937793.026426] [Fri Apr 12 11:14:31 2019][2937793.028122] cpu#37, 1996.203 MHz [Fri Apr 12 11:14:31 2019][2937793.031541] .nr_running : 1 [Fri Apr 12 11:14:31 2019][2937793.036339] .load : 1024 [Fri Apr 12 11:14:31 2019][2937793.041398] .nr_switches : 18872264994 [Fri Apr 12 11:14:31 2019][2937793.047066] .nr_load_updates : 1920080773 [Fri Apr 12 11:14:31 2019][2937793.052648] .nr_uninterruptible : 90749 [Fri Apr 12 11:14:31 2019][2937793.057799] .next_balance : 7232.393804 [Fri Apr 12 11:14:31 2019][2937793.063464] .curr->pid : 0 [Fri Apr 12 11:14:31 2019][2937793.068284] .clock : 2937793068.256489 [Fri Apr 12 11:14:31 2019][2937793.074471] .cpu_load[0] : 0 [Fri Apr 12 11:14:31 2019][2937793.079273] .cpu_load[1] : 396 [Fri Apr 12 11:14:31 2019][2937793.084269] .cpu_load[2] : 184 [Fri Apr 12 11:14:31 2019][2937793.089239] .cpu_load[3] : 815 [Fri Apr 12 11:14:31 2019][2937793.094213] .cpu_load[4] : 483 [Fri Apr 12 11:14:31 2019][2937793.099180] .avg_idle : 144420 [Fri Apr 12 11:14:31 2019][2937793.104416] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:31 2019][2937793.109736] [Fri Apr 12 11:14:31 2019][2937793.111423] cfs_rq[37]:/ [Fri Apr 12 11:14:31 2019][2937793.114135] .exec_clock : 0.000000 [Fri Apr 12 11:14:31 2019][2937793.119534] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:31 2019][2937793.124934] .min_vruntime : 125211490.192396 [Fri Apr 12 11:14:31 2019][2937793.131025] .max_vruntime : 0.000001 [Fri Apr 12 11:14:31 2019][2937793.136427] .spread : 0.000000 [Fri Apr 12 11:14:31 2019][2937793.141825] .spread0 : 8366676.467179 [Fri Apr 12 11:14:31 2019][2937793.147744] .nr_spread_over : 0 [Fri Apr 12 11:14:31 2019][2937793.152571] .nr_running : 0 [Fri Apr 12 11:14:31 2019][2937793.157367] .load : 1024 [Fri Apr 12 11:14:31 2019][2937793.162418] .runnable_load_avg : 0 [Fri Apr 12 11:14:31 2019][2937793.167210] .blocked_load_avg : 233 [Fri Apr 12 11:14:31 2019][2937793.172177] .tg_load_avg : 0 [Fri Apr 12 11:14:31 2019][2937793.176968] .tg_load_contrib : 0 [Fri Apr 12 11:14:31 2019][2937793.181762] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:31 2019][2937793.186559] .tg->runnable_avg : 0 [Fri Apr 12 11:14:31 2019][2937793.191356] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:31 2019][2937793.196166] .throttled : 0 [Fri Apr 12 11:14:31 2019][2937793.200958] .throttle_count : 0 [Fri Apr 12 11:14:31 2019][2937793.205754] .avg->runnable_avg_sum : 24816 [Fri Apr 12 11:14:31 2019][2937793.210893] .avg->runnable_avg_period : 46152 [Fri Apr 12 11:14:31 2019][2937793.216041] [Fri Apr 12 11:14:31 2019][2937793.217711] rt_rq[37]:/ [Fri Apr 12 11:14:31 2019][2937793.220338] .rt_nr_running : 0 [Fri Apr 12 11:14:31 2019][2937793.225130] .rt_throttled : 0 [Fri Apr 12 11:14:31 2019][2937793.229925] .rt_time : 0.000000 [Fri Apr 12 11:14:31 2019][2937793.235324] .rt_runtime : 950.000000 [Fri Apr 12 11:14:31 2019][2937793.240897] [Fri Apr 12 11:14:31 2019][2937793.242568] runnable tasks: [Fri Apr 12 11:14:31 2019][2937793.245541] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:31 2019][2937793.256497] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:31 2019][2937793.267382] watchdog/37 196 -11.986935 734434 0 0.000000 36974.993949 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.278736] migration/37 197 0.000000 155006 0 0.000000 376853.638084 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.290072] ksoftirqd/37 198 125211506.749567 15189800 120 0.000000 317624.997904 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.301406] kworker/37:0H 200 13304.024510 10 100 0.000000 0.036166 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.312762] ccp-1-q4 12395 3595.378970 2 120 0.000000 0.010400 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.324104] multipathd 14477 14547.832430 40 120 0.000000 1.244245 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.335451] multipathd 14512 0.000000 44 0 0.000000 2.623320 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.346811] gssproxy 18070 12759.686892 1 120 0.000000 0.035638 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.358168] gssproxy 18074 12771.694360 1 120 0.000000 0.007474 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.369501] gssproxy 18075 12783.701227 1 120 0.000000 0.006873 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.380842] kworker/37:1H 18358 125211507.581288 10638237 100 0.000000 77464.634627 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.392211] ptlrpcd_01_11 117157 125207525.423666 674273 120 0.000000 15892.556780 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.403638] ldlm_cn01_000 117269 125211452.877948 54143072 120 0.000000 1326465.922806 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.415065] ldlm_cn01_002 117271 125211452.872898 49378499 120 0.000000 1245004.603053 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.426512] mdt01_004 117534 125211525.050386 31923556 120 0.000000 1956765.983123 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.437960] ldlm_cn01_004 117682 125211454.446293 54145505 120 0.000000 1329450.799799 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.449387] ldlm_cn01_005 117687 125211513.546504 51690381 120 0.000000 1295055.838747 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.460853] mdt01_044 118269 125211452.872828 25189880 120 0.000000 1754767.722500 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.472294] mdt01_050 118279 110188341.081493 111642 120 0.000000 4438.663427 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.483726] mdt01_057 118306 125211532.557688 24791385 120 0.000000 1731184.314388 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.495154] mdt01_085 118352 125075168.328880 33358375 120 0.000000 1988464.591546 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.506578] mdt01_087 118354 110178590.710463 3354598 120 0.000000 122778.527551 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.518004] mdt01_094 118376 125211544.019646 24751883 120 0.000000 1649320.581751 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.529497] mdt_io01_060 118969 111120463.318850 69 120 0.000000 3.301150 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.540931] mdt_io01_064 118997 125211294.080953 247340 120 0.000000 20848.669373 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.552379] mdt_io01_082 119072 125211539.395127 2154256 120 0.000000 153506.337854 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.563813] mdt_rdpg01_030 119110 125210869.369983 21436839 120 0.000000 1156414.248295 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.575482] mdt_rdpg01_033 119132 125210869.370654 27949661 120 0.000000 1291076.386458 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.587153] mdt_rdpg01_043 119183 125211345.643300 29299944 120 0.000000 1450706.712976 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.598581] R mdt_rdpg01_047 119197 125211558.359740 23901848 120 0.000000 1280971.755817 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.610017] mdt_io01_105 119200 125211539.683714 1827718 120 0.000000 171614.055947 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.621452] mdt_io01_111 119232 125209724.731440 2286304 120 0.000000 181810.749749 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.632887] kworker/37:2 123397 125211507.207068 247032 120 0.000000 11845.831409 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.644317] ldlm_cn01_025 126535 125211454.444148 46949010 120 0.000000 1200568.083651 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.655771] kworker/37:0 26717 116698545.093940 2 120 0.000000 0.009529 0.000000 1 / [Fri Apr 12 11:14:31 2019][2937793.667133] [Fri Apr 12 11:14:31 2019][2937793.668808] cpu#38, 1996.203 MHz [Fri Apr 12 11:14:31 2019][2937793.672216] .nr_running : 4 [Fri Apr 12 11:14:31 2019][2937793.677018] .load : 0 [Fri Apr 12 11:14:31 2019][2937793.681819] .nr_switches : 8883808710 [Fri Apr 12 11:14:31 2019][2937793.687401] .nr_load_updates : 1690624543 [Fri Apr 12 11:14:31 2019][2937793.692981] .nr_uninterruptible : 52793 [Fri Apr 12 11:14:31 2019][2937793.698121] .next_balance : 7232.394445 [Fri Apr 12 11:14:31 2019][2937793.703817] .curr->pid : 0 [Fri Apr 12 11:14:31 2019][2937793.708617] .clock : 2937793708.070946 [Fri Apr 12 11:14:31 2019][2937793.714813] .cpu_load[0] : 0 [Fri Apr 12 11:14:31 2019][2937793.719619] .cpu_load[1] : 128 [Fri Apr 12 11:14:31 2019][2937793.724610] .cpu_load[2] : 228 [Fri Apr 12 11:14:31 2019][2937793.729610] .cpu_load[3] : 92 [Fri Apr 12 11:14:31 2019][2937793.734496] .cpu_load[4] : 147 [Fri Apr 12 11:14:31 2019][2937793.739471] .avg_idle : 259929 [Fri Apr 12 11:14:31 2019][2937793.744708] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:31 2019][2937793.750028] [Fri Apr 12 11:14:31 2019][2937793.751705] cfs_rq[38]:/ [Fri Apr 12 11:14:31 2019][2937793.754418] .exec_clock : 0.000000 [Fri Apr 12 11:14:31 2019][2937793.759816] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:31 2019][2937793.765216] .min_vruntime : 84047760.637946 [Fri Apr 12 11:14:31 2019][2937793.771223] .max_vruntime : 0.000001 [Fri Apr 12 11:14:31 2019][2937793.776622] .spread : 0.000000 [Fri Apr 12 11:14:31 2019][2937793.782031] .spread0 : -32797120.549983 [Fri Apr 12 11:14:31 2019][2937793.788123] .nr_spread_over : 0 [Fri Apr 12 11:14:31 2019][2937793.792914] .nr_running : 0 [Fri Apr 12 11:14:31 2019][2937793.797707] .load : 0 [Fri Apr 12 11:14:31 2019][2937793.802500] .runnable_load_avg : 0 [Fri Apr 12 11:14:31 2019][2937793.807294] .blocked_load_avg : 163 [Fri Apr 12 11:14:31 2019][2937793.812259] .tg_load_avg : 0 [Fri Apr 12 11:14:31 2019][2937793.817052] .tg_load_contrib : 0 [Fri Apr 12 11:14:31 2019][2937793.821844] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:31 2019][2937793.826639] .tg->runnable_avg : 0 [Fri Apr 12 11:14:31 2019][2937793.831430] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:31 2019][2937793.836222] .throttled : 0 [Fri Apr 12 11:14:31 2019][2937793.841015] .throttle_count : 0 [Fri Apr 12 11:14:31 2019][2937793.845810] .avg->runnable_avg_sum : 5749 [Fri Apr 12 11:14:31 2019][2937793.850860] .avg->runnable_avg_period : 46557 [Fri Apr 12 11:14:31 2019][2937793.856009] [Fri Apr 12 11:14:31 2019][2937793.857683] rt_rq[38]:/ [Fri Apr 12 11:14:31 2019][2937793.860308] .rt_nr_running : 0 [Fri Apr 12 11:14:31 2019][2937793.865100] .rt_throttled : 0 [Fri Apr 12 11:14:32 2019][2937793.869892] .rt_time : 0.000000 [Fri Apr 12 11:14:32 2019][2937793.875293] .rt_runtime : 950.000000 [Fri Apr 12 11:14:32 2019][2937793.880865] [Fri Apr 12 11:14:32 2019][2937793.882538] runnable tasks: [Fri Apr 12 11:14:32 2019][2937793.885511] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:32 2019][2937793.896457] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:32 2019][2937793.907352] watchdog/38 201 -11.985092 734435 0 0.000000 20306.660180 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937793.918696] migration/38 202 0.000000 106631 0 0.000000 270000.004511 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937793.930033] ksoftirqd/38 203 84047748.137893 14198312 120 0.000000 437970.018560 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937793.941369] kworker/38:0H 205 16030.593925 10 100 0.000000 0.040867 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937793.952734] scsi_tmf_0 1957 766.280513 2 100 0.000000 0.013755 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937793.964077] ext4-rsv-conver 7794 2808.395606 2 100 0.000000 0.007945 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937793.975435] mlx4 18210 12374.833571 2 100 0.000000 0.011521 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937793.986792] mlx4_ib 18223 12413.483005 2 100 0.000000 0.012133 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937793.998127] mlx4_ib_mcg 18224 12425.491974 2 100 0.000000 0.009738 0.000000[2937794.007953] LustreError: dumping log to /tmp/lustre-log.1555092872.118290 [Fri Apr 12 11:14:32 2019] 2 / [Fri Apr 12 11:14:32 2019][2937794.016445] rdma_cm 18728 13063.841703 2 100 0.000000 0.011392 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.027801] reader#3 19299 84047022.928613 356895 120 0.000000 38148.201941 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.039160] slapd 19732 83341326.660572 1704 120 0.000000 28.988070 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.050494] kworker/38:1H 19844 84047766.396010 6516661 100 0.000000 36044.201341 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.061844] kworker/38:0 69869 84047705.630839 248830 120 0.000000 8228.619593 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.073186] ptlrpc_hr02_002 117088 84047758.580229 2388127 120 0.000000 75364.241079 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.084606] ptlrpc_hr02_004 117090 84047766.498443 2390289 120 0.000000 76374.477583 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.096036] ptlrpcd_02_04 117162 84047508.202969 835540 120 0.000000 22305.350738 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.107472] mdt_rdpg02_001 117480 84041212.818867 23932394 120 0.000000 1205805.131062 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.118901] mdt02_006 117535 84030253.077740 13296057 120 0.000000 1056552.086944 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.130330] osp-syn-42-2 117586 84047375.082277 236736 120 0.000000 6622.833876 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.141775] osp-syn-18-2 117596 84047373.938136 238767 120 0.000000 6358.154689 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.153213] osp-syn-17-2 117600 84047766.436465 238809 120 0.000000 6409.299844 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.164639] osp-syn-16-2 117612 84047374.381979 237276 120 0.000000 6475.410079 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.176098] osp-syn-40-2 117626 84047374.579504 236577 120 0.000000 6872.196677 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.187549] mdt_rdpg02_004 118029 84019929.954346 15896048 120 0.000000 819091.270731 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.198975] mdt_rdpg02_005 118085 84019929.954345 21176624 120 0.000000 1010411.429390 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.210431] mdt_io02_006 118456 73927465.590849 65 120 0.000000 5.510833 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.221858] mdt02_078 118507 83984598.941628 16673504 120 0.000000 1179922.958608 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.233278] mdt02_087 118522 84047747.406345 17614204 120 0.000000 1134254.573348 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.244702] mdt02_096 118561 84045355.031543 11186887 120 0.000000 958575.804895 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.256127] mdt_io02_009 118605 73927447.151978 109 120 0.000000 4.524532 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.267557] mdt_io02_029 118684 83998936.181457 631679 120 0.000000 38178.662005 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.279013] mdt_io02_094 118934 84047784.076473 2134220 120 0.000000 179558.042169 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.290437] mdt_io02_095 118939 84047782.443851 508086 120 0.000000 37939.863077 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.301860] mdt_io02_098 118951 83998936.180995 3830871 120 0.000000 291919.989101 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.313295] mdt_io02_103 118975 73928027.915571 135 120 0.000000 9.387003 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.324728] mdt_rdpg02_008 119023 84047747.193642 25586210 120 0.000000 1232646.388812 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.336164] mdt_rdpg02_018 119074 84047749.683108 14648010 120 0.000000 783808.304546 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.347588] mdt_rdpg02_023 119088 84041212.817033 27280110 120 0.000000 1345902.580830 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.359031] mdt_rdpg02_044 119184 84047747.194052 23291747 120 0.000000 1049193.717763 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.370467] mdt_rdpg02_048 119195 84047759.193140 14119155 120 0.000000 740635.506544 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.381902] mdt_rdpg02_053 119226 84041212.819108 17863910 120 0.000000 973697.679925 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.393342] mdt_rdpg02_059 119249 84047769.802926 26301130 120 0.000000 1175026.151650 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.404775] ldlm_cn02_009 121118 84047747.195535 46960062 120 0.000000 1120853.474961 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.416237] ldlm_cn02_010 121492 84047747.194983 42479794 120 0.000000 1048312.511052 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.427661] ldlm_cn02_011 123638 84047747.195254 40710701 120 0.000000 1008139.799101 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.439107] ldlm_cn02_017 129939 84047748.111494 35773430 120 0.000000 925732.933101 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.450532] ldlm_cn02_020 129942 84047747.194844 37682391 120 0.000000 942591.187991 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.461956] kworker/38:2 1680 73908198.402114 2 120 0.000000 0.010249 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.473313] osp-pre-6-1 2453 84047666.637769 27726 120 0.000000 804.018157 0.000000 2 / [Fri Apr 12 11:14:32 2019][2937794.484762] [Fri Apr 12 11:14:32 2019][2937794.486443] cpu#39, 1996.203 MHz [Fri Apr 12 11:14:32 2019][2937794.489858] .nr_running : 0 [Fri Apr 12 11:14:32 2019][2937794.494685] .load : 0 [Fri Apr 12 11:14:32 2019][2937794.499485] .nr_switches : 11515229133 [Fri Apr 12 11:14:32 2019][2937794.505154] .nr_load_updates : 1561990732 [Fri Apr 12 11:14:32 2019][2937794.510745] .nr_uninterruptible : 31100 [Fri Apr 12 11:14:32 2019][2937794.515892] .next_balance : 7232.395261 [Fri Apr 12 11:14:32 2019][2937794.521560] .curr->pid : 0 [Fri Apr 12 11:14:32 2019][2937794.526379] .clock : 2937794525.947645 [Fri Apr 12 11:14:32 2019][2937794.532579] .cpu_load[0] : 0 [Fri Apr 12 11:14:32 2019][2937794.537376] .cpu_load[1] : 0 [Fri Apr 12 11:14:32 2019][2937794.542192] .cpu_load[2] : 451 [Fri Apr 12 11:14:32 2019][2937794.547162] .cpu_load[3] : 219 [Fri Apr 12 11:14:32 2019][2937794.552138] .cpu_load[4] : 199 [Fri Apr 12 11:14:32 2019][2937794.557129] .avg_idle : 454308 [Fri Apr 12 11:14:32 2019][2937794.562364] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:32 2019][2937794.567686] [Fri Apr 12 11:14:32 2019][2937794.569363] cfs_rq[39]:/ [Fri Apr 12 11:14:32 2019][2937794.572075] .exec_clock : 0.000000 [Fri Apr 12 11:14:32 2019][2937794.577476] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:32 2019][2937794.582876] .min_vruntime : 70909683.766242 [Fri Apr 12 11:14:32 2019][2937794.588881] .max_vruntime : 0.000001 [Fri Apr 12 11:14:32 2019][2937794.594279] .spread : 0.000000 [Fri Apr 12 11:14:32 2019][2937794.599678] .spread0 : -45935265.440124 [Fri Apr 12 11:14:32 2019][2937794.605771] .nr_spread_over : 0 [Fri Apr 12 11:14:32 2019][2937794.610565] .nr_running : 0 [Fri Apr 12 11:14:32 2019][2937794.615358] .load : 0 [Fri Apr 12 11:14:32 2019][2937794.620149] .runnable_load_avg : 0 [Fri Apr 12 11:14:32 2019][2937794.624952] .blocked_load_avg : 153 [Fri Apr 12 11:14:32 2019][2937794.629919] .tg_load_avg : 0 [Fri Apr 12 11:14:32 2019][2937794.634719] .tg_load_contrib : 0 [Fri Apr 12 11:14:32 2019][2937794.639653] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:32 2019][2937794.644799] .tg->runnable_avg : 0 [Fri Apr 12 11:14:32 2019][2937794.649653] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:32 2019][2937794.654445] .throttled : 0 [Fri Apr 12 11:14:32 2019][2937794.659237] .throttle_count : 0 [Fri Apr 12 11:14:32 2019][2937794.664032] .avg->runnable_avg_sum : 14300 [Fri Apr 12 11:14:32 2019][2937794.669169] .avg->runnable_avg_period : 46665 [Fri Apr 12 11:14:32 2019][2937794.674316] [Fri Apr 12 11:14:32 2019][2937794.675994] rt_rq[39]:/ [Fri Apr 12 11:14:32 2019][2937794.678616] .rt_nr_running : 0 [Fri Apr 12 11:14:32 2019][2937794.683409] .rt_throttled : 0 [Fri Apr 12 11:14:32 2019][2937794.688202] .rt_time : 0.000000 [Fri Apr 12 11:14:32 2019][2937794.693600] .rt_runtime : 950.000000 [Fri Apr 12 11:14:32 2019][2937794.699173] [Fri Apr 12 11:14:32 2019][2937794.700845] runnable tasks: [Fri Apr 12 11:14:32 2019][2937794.703818] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:32 2019][2937794.714784] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:32 2019][2937794.725669] watchdog/39 206 -11.986614 734434 0 0.000000 10298.406196 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.737005] migration/39 207 0.000000 123226 0 0.000000 27361.894116 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.748341] ksoftirqd/39 208 70909585.680147 4424905 120 0.000000 83257.807075 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.759676] kworker/39:0H 210 11564.004620 9 100 0.000000 0.031360 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.771044] kvm-irqfd-clean 16314 4948.429041 2 100 0.000000 0.015149 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.782412] kworker/39:1H 17527 70909585.665180 2529149 100 0.000000 11781.898923 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.793762] ipoib_flush 18354 14101.520544 2 100 0.000000 0.011442 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.805117] mlx5i 18356 14113.528860 2 100 0.000000 0.009027 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.816454] ipoib_wq 18359 14137.544345 2 100 0.000000 0.009719 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.827816] ptlrpc_hr03_001 117094 70909641.820254 1487468 120 0.000000 27912.953563 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.839248] ptlrpc_hr03_004 117097 70909715.361821 1486695 120 0.000000 27871.585651 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.850721] osp_up1-0 117716 70864956.047873 523 120 0.000000 21.445187 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.862148] osp-syn-26-0 117734 70894427.876163 75770 120 0.000000 2383.777152 0.000000 3 / [Fri Apr 12 11:14:32 2019][2937794.873572] osp-syn-5-0 117756 70894607.703180 74852 120 0.000000 2296.644737 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.885024] mdt03_007 117968 70865931.813546 23438958 120 0.000000 1332632.342529 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.896457] mdt03_010 118193 70909278.412714 17032725 120 0.000000 967919.958581 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.907895] mdt03_014 118230 70909558.644139 15775114 120 0.000000 872727.259325 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.919321] mdt03_015 118234 70563387.289114 20484680 120 0.000000 1224792.591565 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.930758] mdt03_034 118385 70563241.164516 19937741 120 0.000000 1036232.852361 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.942184] mdt03_036 118398 70563387.288673 14181054 120 0.000000 975804.887246 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.953607] mdt03_037 118404 70909742.595193 17160399 120 0.000000 991919.345377 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.965050] mdt03_043 118452 70908839.107785 18662086 120 0.000000 1093204.956951 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.976470] mdt03_046 118461 70563387.288533 22554367 120 0.000000 1237853.823995 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.987904] mdt03_050 118477 70563387.288663 9623464 120 0.000000 732608.682586 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937794.999325] mdt03_051 118481 70563241.164196 17118033 120 0.000000 994920.191950 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.010747] mdt_rdpg03_006 118492 70899608.657582 12900869 120 0.000000 557030.897628 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.022171] mdt_rdpg03_007 118496 70909466.440869 12397515 120 0.000000 504095.515634 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.033601] mdt03_062 118586 70563387.288583 731906 120 0.000000 45442.273842 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.045036] mdt03_066 118603 70563241.164667 13136954 120 0.000000 824222.608085 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.056459] mdt03_068 118616 70908839.106734 1589864 120 0.000000 102839.380923 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.067891] mdt03_072 118636 70908879.060195 7703765 120 0.000000 337890.577066 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.079322] mdt_io03_025 118708 59906370.545041 258118 120 0.000000 10105.734701 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.090744] mdt_io03_028 118720 59906743.019123 303 120 0.000000 12.705467 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.102166] mdt_io03_032 118732 59906395.029738 44 120 0.000000 2.631146 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.113593] mdt03_104 118777 70908839.094379 7903010 120 0.000000 346682.196293 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.125012] mdt03_105 118781 70908839.106734 196577 120 0.000000 12899.690961 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.136437] mdt03_111 118803 70909278.412525 1558345 120 0.000000 96703.040505 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.147861] mdt_io03_050 118810 59906382.861835 154407 120 0.000000 6403.379508 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.159281] mdt_io03_051 118813 59906395.340813 370573 120 0.000000 13921.786241 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.170770] mdt_rdpg03_041 119231 70903975.483366 14242035 120 0.000000 556877.640653 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.182195] mdt_rdpg03_042 119238 70903975.483084 13767374 120 0.000000 572263.768579 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.193619] mdt_rdpg03_043 119240 70903975.482403 11885305 120 0.000000 507887.089510 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.205045] mdt_rdpg03_050 119258 70909466.440970 16249923 120 0.000000 655963.247412 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.216463] mdt_rdpg03_052 119260 70903975.482644 15779738 120 0.000000 647791.674548 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.227887] mdt_rdpg03_058 119266 70909466.681817 14253400 120 0.000000 626908.978143 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.239327] ldlm_cn03_015 129486 70909466.445377 41272829 120 0.000000 807056.911672 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.250766] osp-syn-28-3 2322 70905622.103776 9499 120 0.000000 277.543716 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.262104] osp_up0-1 2384 70874449.137389 410 120 0.000000 17.009540 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.273451] osp-syn-5-1 2426 70904891.884941 19923 120 0.000000 491.219855 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.284787] osp-syn-29-1 2462 70905376.275259 20194 120 0.000000 505.023393 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.296123] osp-syn-37-1 2472 70904867.739232 19969 120 0.000000 499.618240 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.307460] mdt_io03_092 3062 59906382.768258 9 120 0.000000 0.393129 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.318802] kworker/39:1 27263 70909756.541811 100722 120 0.000000 2361.308552 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.330144] ldlm_cn03_021 44378 70908809.135036 1033052 120 0.000000 29301.295698 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.341476] ldlm_cn03_026 44384 70908809.134824 1348590 120 0.000000 36710.353612 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.352811] kworker/39:2 44858 70382926.360567 3 120 0.000000 0.011712 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.364149] ldlm_cn03_028 45020 70909466.517305 954390 120 0.000000 27089.332680 0.000000 3 / [Fri Apr 12 11:14:33 2019][2937795.375558] [Fri Apr 12 11:14:33 2019][2937795.377237] cpu#40, 1996.203 MHz [Fri Apr 12 11:14:33 2019][2937795.380646] .nr_running : 0 [Fri Apr 12 11:14:33 2019][2937795.385446] .load : 1024 [Fri Apr 12 11:14:33 2019][2937795.390499] .nr_switches : 20935119134 [Fri Apr 12 11:14:33 2019][2937795.396160] .nr_load_updates : 1911846548 [Fri Apr 12 11:14:33 2019][2937795.401738] .nr_uninterruptible : -676652 [Fri Apr 12 11:14:33 2019][2937795.407053] .next_balance : 7232.396152 [Fri Apr 12 11:14:33 2019][2937795.412713] .curr->pid : 118558 [Fri Apr 12 11:14:33 2019][2937795.417947] .clock : 2937795417.921457 [Fri Apr 12 11:14:33 2019][2937795.424126] .cpu_load[0] : 0 [Fri Apr 12 11:14:33 2019][2937795.428919] .cpu_load[1] : 13 [Fri Apr 12 11:14:33 2019][2937795.433798] .cpu_load[2] : 30 [Fri Apr 12 11:14:33 2019][2937795.438676] .cpu_load[3] : 63 [Fri Apr 12 11:14:33 2019][2937795.443557] .cpu_load[4] : 455 [Fri Apr 12 11:14:33 2019][2937795.448523] .avg_idle : 676004 [Fri Apr 12 11:14:33 2019][2937795.453750] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:33 2019][2937795.459063] [Fri Apr 12 11:14:33 2019][2937795.460741] cfs_rq[40]:/ [Fri Apr 12 11:14:33 2019][2937795.463454] .exec_clock : 0.000000 [Fri Apr 12 11:14:33 2019][2937795.468852] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:33 2019][2937795.474251] .min_vruntime : 113565619.716951 [Fri Apr 12 11:14:33 2019][2937795.480345] .max_vruntime : 0.000001 [Fri Apr 12 11:14:33 2019][2937795.485742] .spread : 0.000000 [Fri Apr 12 11:14:33 2019][2937795.491145] .spread0 : -3279416.381539 [Fri Apr 12 11:14:33 2019][2937795.497150] .nr_spread_over : 0 [Fri Apr 12 11:14:33 2019][2937795.501941] .nr_running : 1 [Fri Apr 12 11:14:33 2019][2937795.506736] .load : 1024 [Fri Apr 12 11:14:33 2019][2937795.511787] .runnable_load_avg : 312 [Fri Apr 12 11:14:33 2019][2937795.516754] .blocked_load_avg : 287 [Fri Apr 12 11:14:33 2019][2937795.521721] .tg_load_avg : 0 [Fri Apr 12 11:14:33 2019][2937795.526512] .tg_load_contrib : 0 [Fri Apr 12 11:14:33 2019][2937795.531305] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:33 2019][2937795.536099] .tg->runnable_avg : 0 [Fri Apr 12 11:14:33 2019][2937795.540891] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:33 2019][2937795.545684] .throttled : 0 [Fri Apr 12 11:14:33 2019][2937795.550476] .throttle_count : 0 [Fri Apr 12 11:14:33 2019][2937795.555270] .avg->runnable_avg_sum : 12640 [Fri Apr 12 11:14:33 2019][2937795.560409] .avg->runnable_avg_period : 46911 [Fri Apr 12 11:14:33 2019][2937795.565555] [Fri Apr 12 11:14:33 2019][2937795.567229] rt_rq[40]:/ [Fri Apr 12 11:14:33 2019][2937795.569855] .rt_nr_running : 0 [Fri Apr 12 11:14:33 2019][2937795.574647] .rt_throttled : 0 [Fri Apr 12 11:14:33 2019][2937795.579440] .rt_time : 0.000000 [Fri Apr 12 11:14:33 2019][2937795.584841] .rt_runtime : 950.000000 [Fri Apr 12 11:14:33 2019][2937795.590412] [Fri Apr 12 11:14:33 2019][2937795.592087] runnable tasks: [Fri Apr 12 11:14:33 2019][2937795.595060] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:33 2019][2937795.606004] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:33 2019][2937795.616891] watchdog/40 211 -11.988087 734435 0 0.000000 25879.600541 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.628227] migration/40 212 0.000000 20256 0 0.000000 219664.236104 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.639563] ksoftirqd/40 213 113565491.638258 18369880 120 0.000000 264558.482414 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.650900] kworker/40:0H 215 6364.941664 10 100 0.000000 0.046940 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.662273] smartd 18024 113564711.994084 18158 120 0.000000 642.028775 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.673616] kworker/40:1H 20078 113565491.626137 15381967 100 0.000000 105452.971558 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.684987] kworker/40:2 55670 113564711.756113 701425 120 0.000000 14563.918204 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.696348] ldlm_cn00_000 117266 113565488.227581 41551010 120 0.000000 979311.275047 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.707773] jbd2/dm-3-8 117447 113565436.048858 401349 120 0.000000 2521473.148998 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.719226] osp-pre-28-2 117659 113565235.782691 40684 120 0.000000 1087.621719 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.730661] osp-syn-9-0 117740 113533647.979125 75122 120 0.000000 2426.371375 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.742101] mdt_rdpg00_002 117970 113565488.222642 18507129 120 0.000000 820292.269659 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.753531] mdt00_018 118181 113560175.540463 9875862 120 0.000000 774574.453160 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.764960] mdt00_026 118211 113559005.708429 40932321 120 0.000000 2154875.609659 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.776380] mdt00_031 118224 108105559.375720 23498673 120 0.000000 1130493.964373 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.787803] mdt00_044 118263 113521465.031147 21633631 120 0.000000 1226734.480528 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.799227] mdt00_056 118303 97237570.637989 3726133 120 0.000000 129311.397837 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.810648] mdt00_060 118314 113564082.941759 17592120 120 0.000000 1113640.713223 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.822076] mdt00_063 118363 113565674.854663 27107625 120 0.000000 1389057.426514 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.833502] mdt00_065 118380 113485425.946810 37852019 120 0.000000 1937379.941378 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.844925] mdt00_067 118386 113565667.550965 32435130 120 0.000000 1862375.645494 0.000000 0 / [Fri Apr 12 11:14:33 2019][2937795.856349] mdt00_069 118396 113564081.922666 28125763 120 0.000000 1638348.512812 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.867776] mdt00_085 118459 113565684.131701 30775868 120 0.000000 1547466.673495 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.879208] mdt00_093 118536 113564067.994924 43453911 120 0.000000 2039410.360013 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.890633] mdt00_094 118537 113563603.780424 35785586 120 0.000000 1751847.252456 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.902058] mdt00_098 118552 112501640.717862 31172396 120 0.000000 1596203.450430 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.913481] mdt00_105 118560 111962512.742468 27145254 120 0.000000 1451882.887121 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.924911] mdt00_108 118567 113565681.437348 34535152 120 0.000000 1758672.454979 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.936337] mdt_io00_007 118599 98531704.458176 269672 120 0.000000 12531.640297 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.947757] mdt_io00_011 118611 98531498.428100 104 120 0.000000 5.981725 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.959182] mdt_io00_015 118624 98531703.752498 108 120 0.000000 5.158797 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.970612] mdt_io00_030 118691 98531703.948109 1601 120 0.000000 46.399278 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.982045] mdt_io00_047 118804 98531497.524305 71 120 0.000000 3.490358 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937795.993477] mdt_io00_059 118864 98531703.628682 253317 120 0.000000 8130.790837 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.004903] mdt_io00_064 118889 98531498.815145 50 120 0.000000 3.504392 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.016330] mdt_io00_080 118961 98531497.871926 41 120 0.000000 2.379127 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.027776] mdt_io00_107 119115 98531497.362817 232276 120 0.000000 10252.702164 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.039204] mdt_rdpg00_028 119128 113564491.355075 21830517 120 0.000000 991570.198936 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.050636] mdt_rdpg00_047 119214 113564844.820395 21290278 120 0.000000 916710.349293 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.062068] ldlm_cn00_008 119915 113565690.166807 42838951 120 0.000000 1021748.626717 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.073489] ldlm_cn00_010 121722 113565488.222152 41996250 120 0.000000 1007212.935319 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.084919] ldlm_cn00_016 127039 113565686.215477 38071754 120 0.000000 930109.977366 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.096351] osp-pre-31-3 2249 113565138.784801 28028 120 0.000000 616.674676 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.107703] ldlm_cn00_026 6483 113565488.222351 30200932 120 0.000000 752911.091410 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.119041] ldlm_cn00_029 6486 113565675.841285 33140623 120 0.000000 808880.268240 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.130380] kworker/40:0 26850 113564724.470075 17 120 0.000000 0.138865 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.141723] kworker/40:1 46735 113565688.896233 26 120 0.000000 0.612862 0.000000 0 / [Fri Apr 12 11:14:34 2019][2937796.153121] [Fri Apr 12 11:14:34 2019][2937796.154792] cpu#41, 1996.203 MHz [Fri Apr 12 11:14:34 2019][2937796.158202] .nr_running : 0 [Fri Apr 12 11:14:34 2019][2937796.163002] .load : 0 [Fri Apr 12 11:14:34 2019][2937796.167796] .nr_switches : 22134511471 [Fri Apr 12 11:14:34 2019][2937796.173455] .nr_load_updates : 2002688304 [Fri Apr 12 11:14:34 2019][2937796.179041] .nr_uninterruptible : 72626 [Fri Apr 12 11:14:34 2019][2937796.184184] .next_balance : 7232.396929 [Fri Apr 12 11:14:34 2019][2937796.189843] .curr->pid : 117122 [Fri Apr 12 11:14:34 2019][2937796.195070] .clock : 2937796194.983359 [Fri Apr 12 11:14:34 2019][2937796.201250] .cpu_load[0] : 0 [Fri Apr 12 11:14:34 2019][2937796.206050] .cpu_load[1] : 1334 [Fri Apr 12 11:14:34 2019][2937796.211102] .cpu_load[2] : 633 [Fri Apr 12 11:14:34 2019][2937796.216069] .cpu_load[3] : 925 [Fri Apr 12 11:14:34 2019][2937796.221038] .cpu_load[4] : 641 [Fri Apr 12 11:14:34 2019][2937796.226011] .avg_idle : 53298 [Fri Apr 12 11:14:34 2019][2937796.231149] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:34 2019][2937796.236462] [Fri Apr 12 11:14:34 2019][2937796.238133] cfs_rq[41]:/ [Fri Apr 12 11:14:34 2019][2937796.240845] .exec_clock : 0.000000 [Fri Apr 12 11:14:34 2019][2937796.246246] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:34 2019][2937796.251644] .min_vruntime : 128600929.632415 [Fri Apr 12 11:14:34 2019][2937796.257738] .max_vruntime : 0.000001 [Fri Apr 12 11:14:34 2019][2937796.263136] .spread : 0.000000 [Fri Apr 12 11:14:34 2019][2937796.268536] .spread0 : 11755814.312526 [Fri Apr 12 11:14:34 2019][2937796.274540] .nr_spread_over : 0 [Fri Apr 12 11:14:34 2019][2937796.279335] .nr_running : 1 [Fri Apr 12 11:14:34 2019][2937796.284126] .load : 1024 [Fri Apr 12 11:14:34 2019][2937796.289180] .runnable_load_avg : 0 [Fri Apr 12 11:14:34 2019][2937796.293972] .blocked_load_avg : 450 [Fri Apr 12 11:14:34 2019][2937796.298938] .tg_load_avg : 0 [Fri Apr 12 11:14:34 2019][2937796.303732] .tg_load_contrib : 0 [Fri Apr 12 11:14:34 2019][2937796.308523] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:34 2019][2937796.313317] .tg->runnable_avg : 0 [Fri Apr 12 11:14:34 2019][2937796.318109] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:34 2019][2937796.322902] .throttled : 0 [Fri Apr 12 11:14:34 2019][2937796.327696] .throttle_count : 0 [Fri Apr 12 11:14:34 2019][2937796.332488] .avg->runnable_avg_sum : 13980 [Fri Apr 12 11:14:34 2019][2937796.337628] .avg->runnable_avg_period : 46148 [Fri Apr 12 11:14:34 2019][2937796.342777] [Fri Apr 12 11:14:34 2019][2937796.344447] rt_rq[41]:/ [Fri Apr 12 11:14:34 2019][2937796.347075] .rt_nr_running : 0 [Fri Apr 12 11:14:34 2019][2937796.351866] .rt_throttled : 0 [Fri Apr 12 11:14:34 2019][2937796.356660] .rt_time : 0.000000 [Fri Apr 12 11:14:34 2019][2937796.362059] .rt_runtime : 1000.000000 [Fri Apr 12 11:14:34 2019][2937796.367719] [Fri Apr 12 11:14:34 2019][2937796.369391] runnable tasks: [Fri Apr 12 11:14:34 2019][2937796.372365] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:34 2019][2937796.383310] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:34 2019][2937796.394198] watchdog/41 216 -11.986064 734435 0 0.000000 38998.357911 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.405532] migration/41 217 0.000000 29702 0 0.000000 341735.151943 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.416868] ksoftirqd/41 218 128600928.360489 12456972 120 0.000000 295633.975191 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.428204] kworker/41:0H 220 4919.204297 10 100 0.000000 0.054071 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.439576] mdadm 18063 128517196.298053 2939 120 0.000000 293.986287 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.450921] kworker/41:1H 21302 128600928.342505 8243543 100 0.000000 65659.755215 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.462260] kworker/41:2 19904 128600917.265835 1765484 120 0.000000 59005.308953 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.473599] kworker/41:0 113295 98183699.835730 2 120 0.000000 0.009719 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.485031] Rkiblnd_sd_01_02 117121 128600984.815298 2224668513 120 0.000000 21817682.668088 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.496563] mdt_io01_002 117512 118399523.869110 53308 120 0.000000 2319.305873 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.508047] mdt01_045 118271 128542374.334209 29917413 120 0.000000 1946954.630733 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.519474] mdt01_074 118337 128493154.433404 30386216 120 0.000000 1883108.746792 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.530898] mdt01_075 118338 128491798.053040 20970295 120 0.000000 1502088.222104 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.542363] mdt_io01_027 118765 118399523.868209 2292 120 0.000000 89.529461 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.553794] mdt_io01_031 118819 118399523.868640 475711 120 0.000000 28278.105020 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.565218] mdt_io01_032 118822 118399523.868679 44541 120 0.000000 2137.929784 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.576644] mdt_io01_036 118831 118399523.867678 9700 120 0.000000 452.997488 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.588069] mdt_io01_042 118850 118399523.867308 85250 120 0.000000 3825.431672 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.599499] mdt_io01_050 118914 118399532.707370 385971 120 0.000000 20778.669118 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.610932] mdt_rdpg01_007 119024 128598996.694494 22666204 120 0.000000 1145020.404636 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.622361] mdt_rdpg01_021 119087 128592061.339587 31985129 120 0.000000 1419051.122819 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.633784] mdt_rdpg01_026 119100 128592061.339367 29706562 120 0.000000 1376634.224018 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.645206] mdt_rdpg01_027 119102 128600647.451221 27019689 120 0.000000 1292439.829697 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.656631] mdt_io01_090 119118 118399527.391406 801782 120 0.000000 31761.283960 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.668053] mdt_io01_091 119120 118399527.811802 779320 120 0.000000 32090.850452 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.679492] mdt_rdpg01_037 119170 128588506.606499 27579718 120 0.000000 1383320.230455 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.690917] mdt_rdpg01_045 119190 128592061.339477 26916746 120 0.000000 1326244.658358 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.702344] mdt_rdpg01_058 119243 128592061.339336 23425671 120 0.000000 1207563.074256 0.000000 1 / [Fri Apr 12 11:14:34 2019][2937796.713850] [Fri Apr 12 11:14:34 2019][2937796.715524] cpu#42, 1996.203 MHz [Fri Apr 12 11:14:34 2019][2937796.718940] .nr_running : 1 [Fri Apr 12 11:14:34 2019][2937796.723742] .load : 0 [Fri Apr 12 11:14:34 2019][2937796.728544] .nr_switches : 11931325984 [Fri Apr 12 11:14:34 2019][2937796.734209] .nr_load_updates : 1768652141 [Fri Apr 12 11:14:34 2019][2937796.739782] .nr_uninterruptible : 124608 [Fri Apr 12 11:14:34 2019][2937796.745009] .next_balance : 7232.397490 [Fri Apr 12 11:14:34 2019][2937796.750669] .curr->pid : 0 [Fri Apr 12 11:14:34 2019][2937796.755488] .clock : 2937796755.409207 [Fri Apr 12 11:14:34 2019][2937796.761675] .cpu_load[0] : 0 [Fri Apr 12 11:14:34 2019][2937796.766467] .cpu_load[1] : 1009 [Fri Apr 12 11:14:34 2019][2937796.771520] .cpu_load[2] : 659 [Fri Apr 12 11:14:34 2019][2937796.776487] .cpu_load[3] : 348 [Fri Apr 12 11:14:34 2019][2937796.781453] .cpu_load[4] : 409 [Fri Apr 12 11:14:34 2019][2937796.786419] .avg_idle : 78558 [Fri Apr 12 11:14:34 2019][2937796.791557] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:34 2019][2937796.796871] [Fri Apr 12 11:14:34 2019][2937796.798540] cfs_rq[42]:/ [Fri Apr 12 11:14:34 2019][2937796.801254] .exec_clock : 0.000000 [Fri Apr 12 11:14:34 2019][2937796.806653] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:34 2019][2937796.812051] .min_vruntime : 96829911.354945 [Fri Apr 12 11:14:34 2019][2937796.818057] .max_vruntime : 0.000001 [Fri Apr 12 11:14:34 2019][2937796.823457] .spread : 0.000000 [Fri Apr 12 11:14:34 2019][2937796.828857] .spread0 : -20015262.103721 [Fri Apr 12 11:14:34 2019][2937796.834951] .nr_spread_over : 0 [Fri Apr 12 11:14:34 2019][2937796.839742] .nr_running : 0 [Fri Apr 12 11:14:34 2019][2937796.844535] .load : 0 [Fri Apr 12 11:14:34 2019][2937796.849327] .runnable_load_avg : 0 [Fri Apr 12 11:14:34 2019][2937796.854119] .blocked_load_avg : 282 [Fri Apr 12 11:14:34 2019][2937796.859087] .tg_load_avg : 0 [Fri Apr 12 11:14:34 2019][2937796.863879] .tg_load_contrib : 0 [Fri Apr 12 11:14:35 2019][2937796.868671] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:35 2019][2937796.873464] .tg->runnable_avg : 0 [Fri Apr 12 11:14:35 2019][2937796.878258] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:35 2019][2937796.883050] .throttled : 0 [Fri Apr 12 11:14:35 2019][2937796.887843] .throttle_count : 0 [Fri Apr 12 11:14:35 2019][2937796.892638] .avg->runnable_avg_sum : 15728 [Fri Apr 12 11:14:35 2019][2937796.897777] .avg->runnable_avg_period : 46458 [Fri Apr 12 11:14:35 2019][2937796.902923] [Fri Apr 12 11:14:35 2019][2937796.904596] rt_rq[42]:/ [Fri Apr 12 11:14:35 2019][2937796.907222] .rt_nr_running : 0 [Fri Apr 12 11:14:35 2019][2937796.912016] .rt_throttled : 0 [Fri Apr 12 11:14:35 2019][2937796.916808] .rt_time : 0.000000 [Fri Apr 12 11:14:35 2019][2937796.922207] .rt_runtime : 950.000000 [Fri Apr 12 11:14:35 2019][2937796.927779] [Fri Apr 12 11:14:35 2019][2937796.929452] runnable tasks: [Fri Apr 12 11:14:35 2019][2937796.932424] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:35 2019][2937796.943371] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:35 2019][2937796.954260] watchdog/42 221 -11.987206 734435 0 0.000000 29849.725697 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937796.965593] migration/42 222 0.000000 23522 0 0.000000 297644.083233 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937796.976928] ksoftirqd/42 223 96829943.276827 22593444 120 0.000000 691709.967139 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937796.988267] kworker/42:0H 225 5465.972972 11 100 0.000000 0.077495 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937796.999637] polkitd 18048 4894.313460 1 120 0.000000 0.035156 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.010974] JS GC Helper 18049 4906.385341 4 120 0.000000 0.071887 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.022318] writer#4 19295 96829940.613530 15199361 120 0.000000 236195.528190 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.033663] kworker/42:1H 20877 96829405.779896 8439391 100 0.000000 40379.906144 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.045009] kworker/42:1 12111 96829800.424640 842620 120 0.000000 18107.055704 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.056364] kiblnd_sd_02_01 117124 138968283.959134 1947576843 120 0.000000 16170405.969575 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.067871] kiblnd_sd_02_02 117125 138968285.481293 1947520821 120 0.000000 16167068.319436 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.079396] mdt_rdpg02_000 117479 96790257.624449 20516544 120 0.000000 961203.578683 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.090882] mdt02_010 118161 96788553.742692 16336570 120 0.000000 1139069.811667 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.102313] mdt02_021 118209 96817003.985086 12657895 120 0.000000 993354.187873 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.113744] mdt02_034 118293 81098371.080498 6022697 120 0.000000 232919.068900 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.125172] mdt02_041 118347 96829944.035383 21959348 120 0.000000 1318039.271132 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.136600] mdt_io02_005 118444 82133802.257796 130 120 0.000000 7.632865 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.148035] mdt02_104 118582 96817003.990686 14503866 120 0.000000 1195247.799381 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.159457] mdt02_111 118592 96825773.826427 13037723 120 0.000000 1155814.817494 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.170880] mdt_io02_015 118634 82133619.641178 78 120 0.000000 4.188145 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.182342] mdt_io02_097 118949 82133788.507042 138 120 0.000000 5.601802 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.193772] mdt_io02_104 118976 82133814.819561 140 120 0.000000 5.706394 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.205198] mdt_io02_106 118978 82133594.923217 58 120 0.000000 2.089545 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.216635] mdt_rdpg02_019 119079 96811997.680371 17389424 120 0.000000 956241.283979 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.228083] mdt_rdpg02_051 119220 96790655.986124 27129568 120 0.000000 1293687.051365 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.239560] kworker/42:0 16497 84098090.608317 2 120 0.000000 0.006042 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.250910] sshd 46542 96779808.856877 10980 120 0.000000 237.397749 0.000000 2 / [Fri Apr 12 11:14:35 2019][2937797.262283] [Fri Apr 12 11:14:35 2019][2937797.263964] cpu#43, 1996.203 MHz [Fri Apr 12 11:14:35 2019][2937797.267378] .nr_running : 0 [Fri Apr 12 11:14:35 2019][2937797.272171] .load : 0 [Fri Apr 12 11:14:35 2019][2937797.276973] .nr_switches : 9524669710 [Fri Apr 12 11:14:35 2019][2937797.282556] .nr_load_updates : 1548901535 [Fri Apr 12 11:14:35 2019][2937797.288134] .nr_uninterruptible : 30128 [Fri Apr 12 11:14:35 2019][2937797.293274] .next_balance : 7232.398040 [Fri Apr 12 11:14:35 2019][2937797.298935] .curr->pid : 0 [Fri Apr 12 11:14:35 2019][2937797.303738] .clock : 2937797303.735935 [Fri Apr 12 11:14:35 2019][2937797.309924] .cpu_load[0] : 0 [Fri Apr 12 11:14:35 2019][2937797.314727] .cpu_load[1] : 8 [Fri Apr 12 11:14:35 2019][2937797.319528] .cpu_load[2] : 9 [Fri Apr 12 11:14:35 2019][2937797.324331] .cpu_load[3] : 13 [Fri Apr 12 11:14:35 2019][2937797.329217] .cpu_load[4] : 88 [Fri Apr 12 11:14:35 2019][2937797.334096] .avg_idle : 250658 [Fri Apr 12 11:14:35 2019][2937797.339331] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:35 2019][2937797.344654] [Fri Apr 12 11:14:35 2019][2937797.346329] cfs_rq[43]:/ [Fri Apr 12 11:14:35 2019][2937797.349050] .exec_clock : 0.000000 [Fri Apr 12 11:14:35 2019][2937797.354452] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:35 2019][2937797.359910] .min_vruntime : 60765411.565713 [Fri Apr 12 11:14:35 2019][2937797.365917] .max_vruntime : 0.000001 [Fri Apr 12 11:14:35 2019][2937797.371316] .spread : 0.000000 [Fri Apr 12 11:14:35 2019][2937797.376716] .spread0 : -56079813.921182 [Fri Apr 12 11:14:35 2019][2937797.382809] .nr_spread_over : 0 [Fri Apr 12 11:14:35 2019][2937797.387609] .nr_running : 0 [Fri Apr 12 11:14:35 2019][2937797.392404] .load : 1024 [Fri Apr 12 11:14:35 2019][2937797.397455] .runnable_load_avg : 0 [Fri Apr 12 11:14:35 2019][2937797.402248] .blocked_load_avg : 61 [Fri Apr 12 11:14:35 2019][2937797.407128] .tg_load_avg : 0 [Fri Apr 12 11:14:35 2019][2937797.411928] .tg_load_contrib : 0 [Fri Apr 12 11:14:35 2019][2937797.416722] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:35 2019][2937797.421524] .tg->runnable_avg : 0 [Fri Apr 12 11:14:35 2019][2937797.426315] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:35 2019][2937797.431117] .throttled : 0 [Fri Apr 12 11:14:35 2019][2937797.435910] .throttle_count : 0 [Fri Apr 12 11:14:35 2019][2937797.440703] .avg->runnable_avg_sum : 3941 [Fri Apr 12 11:14:35 2019][2937797.445756] .avg->runnable_avg_period : 46492 [Fri Apr 12 11:14:35 2019][2937797.450905] [Fri Apr 12 11:14:35 2019][2937797.452587] rt_rq[43]:/ [Fri Apr 12 11:14:35 2019][2937797.455211] .rt_nr_running : 0 [Fri Apr 12 11:14:35 2019][2937797.460006] .rt_throttled : 0 [Fri Apr 12 11:14:35 2019][2937797.464799] .rt_time : 0.000000 [Fri Apr 12 11:14:35 2019][2937797.470205] .rt_runtime : 950.000000 [Fri Apr 12 11:14:35 2019][2937797.475778] [Fri Apr 12 11:14:35 2019][2937797.477451] runnable tasks: [Fri Apr 12 11:14:35 2019][2937797.480426] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:35 2019][2937797.491370] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:35 2019][2937797.502268] watchdog/43 226 -11.983308 734436 0 0.000000 10336.100449 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.513634] migration/43 227 0.000000 10330 0 0.000000 48393.088885 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.524970] ksoftirqd/43 228 60765379.987281 4604086 120 0.000000 66922.176963 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.536316] kworker/43:0H 230 4635.623920 10 100 0.000000 0.038172 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.547688] gmain 18046 4352.132942 2 120 0.000000 0.088487 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.559032] reader#4 19300 4516.133821 1 120 0.000000 0.007855 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.570537] kworker/43:1H 21060 60765379.963153 3077394 100 0.000000 14924.244025 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.581883] kworker/43:1 58393 60765383.063861 1009833 120 0.000000 15105.333098 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.593241] ptlrpc_hr03_002 117095 60765388.377965 1486944 120 0.000000 27956.508333 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.604665] kiblnd_sd_03_03 117130 60765427.717254 1411338898 120 0.000000 11221689.963794 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.616189] lnet_discovery 117132 60765367.609868 189602 120 0.000000 1513.928565 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.627621] ldlm_cn03_000 117275 60765068.878349 50979728 120 0.000000 969469.271800 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.639045] ldlm_cn03_001 117276 60765068.878198 53608767 120 0.000000 1017368.989899 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.650486] mdt_out03_000 117498 60765271.976695 182059 120 0.000000 5980.964580 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.661916] osp-pre-13-2 117571 60765315.732505 39921 120 0.000000 896.043188 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.673360] osp-pre-1-0 117717 60765241.823868 36806 120 0.000000 722.625774 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.684781] osp-syn-14-0 117732 60750878.629151 75889 120 0.000000 2243.741996 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.696302] osp-syn-20-0 117774 60750828.284370 75470 120 0.000000 2243.246680 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.707743] osp-syn-36-0 117804 60735638.593904 74819 120 0.000000 2243.563849 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.719184] mdt03_011 118197 60764116.011373 21752217 120 0.000000 1134559.320797 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.731075] mdt_rdpg03_013 118511 60756267.504207 13797308 120 0.000000 555120.596330 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.742616] mdt_rdpg03_017 118525 60765361.581441 16682245 120 0.000000 714869.547504 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.754044] mdt_rdpg03_021 118538 60765351.981096 10008501 120 0.000000 394323.507722 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.765472] mdt_rdpg03_028 118590 60765423.490102 15021751 120 0.000000 539753.232554 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.776901] mdt_io03_006 118612 60750596.069450 1577286 120 0.000000 99204.493511 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.788321] mdt_io03_007 118620 51121126.636045 1061 120 0.000000 22.606825 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.799759] mdt_io03_036 118756 51121123.123464 27 120 0.000000 0.988308 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.811187] mdt03_100 118762 60750482.037515 17959566 120 0.000000 922421.955716 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.822609] mdt03_106 118782 54697501.527146 11216664 120 0.000000 606058.074117 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.834035] mdt_io03_048 118805 60765427.488059 1755540 120 0.000000 97915.829026 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.845479] mdt_io03_073 118888 60750596.069920 1304885 120 0.000000 89865.303712 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.856905] mdt_io03_079 118907 51121123.264190 19 120 0.000000 0.701401 0.000000 3 / [Fri Apr 12 11:14:35 2019][2937797.868367] mdt_rdpg03_036 119176 60756267.504237 12632027 120 0.000000 516586.796579 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.879814] mdt_rdpg03_039 119217 60765351.979692 15370262 120 0.000000 665639.828177 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.891241] mdt_rdpg03_040 119224 60749578.278677 14040921 120 0.000000 683075.554343 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.902667] mdt_rdpg03_046 119251 60756267.504948 21127923 120 0.000000 875691.101091 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.914089] mdt_rdpg03_059 119267 60756267.504307 17499276 120 0.000000 736246.596062 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.925518] ldlm_cn03_013 123642 60765068.878398 44263277 120 0.000000 858061.368958 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.936958] ldlm_cn03_016 129487 60765434.906706 41597610 120 0.000000 806616.661043 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.948385] mdt_out03_004 1919 60765395.936719 148577 120 0.000000 5151.772018 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.959729] osp-syn-11-3 2242 60763227.616723 9536 120 0.000000 299.779855 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.971070] osp_up2-1 2381 60738837.450217 408 120 0.000000 20.156485 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.982409] osp-syn-11-1 2400 60760872.528040 19643 120 0.000000 483.099258 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937797.993743] osp-pre-43-1 2415 60765210.215064 27761 120 0.000000 607.729134 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937798.005082] osp-pre-1-1 2465 60765221.818170 28055 120 0.000000 671.758163 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937798.016416] osp-pre-3-1 2469 60765326.965099 27685 120 0.000000 802.072576 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937798.027767] kworker/43:0 39820 58746122.738520 2 120 0.000000 0.008767 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937798.039110] ldlm_cn03_028 45020 60765441.059385 956368 120 0.000000 27131.433783 0.000000 3 / [Fri Apr 12 11:14:36 2019][2937798.050514] [Fri Apr 12 11:14:36 2019][2937798.052197] cpu#44, 1996.203 MHz [Fri Apr 12 11:14:36 2019][2937798.055609] .nr_running : 0 [Fri Apr 12 11:14:36 2019][2937798.060407] .load : 0 [Fri Apr 12 11:14:36 2019][2937798.065216] .nr_switches : 15031351546 [Fri Apr 12 11:14:36 2019][2937798.070883] .nr_load_updates : 1837679356 [Fri Apr 12 11:14:36 2019][2937798.076456] .nr_uninterruptible : -784602 [Fri Apr 12 11:14:36 2019][2937798.081772] .next_balance : 7232.398826 [Fri Apr 12 11:14:36 2019][2937798.087449] .curr->pid : 0 [Fri Apr 12 11:14:36 2019][2937798.092251] .clock : 2937798092.239429 [Fri Apr 12 11:14:36 2019][2937798.098448] .cpu_load[0] : 0 [Fri Apr 12 11:14:36 2019][2937798.103247] .cpu_load[1] : 422 [Fri Apr 12 11:14:36 2019][2937798.108238] .cpu_load[2] : 221 [Fri Apr 12 11:14:36 2019][2937798.113223] .cpu_load[3] : 185 [Fri Apr 12 11:14:36 2019][2937798.118196] .cpu_load[4] : 412 [Fri Apr 12 11:14:36 2019][2937798.123174] .avg_idle : 38731 [Fri Apr 12 11:14:36 2019][2937798.128322] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:36 2019][2937798.133642] [Fri Apr 12 11:14:36 2019][2937798.135318] cfs_rq[44]:/ [Fri Apr 12 11:14:36 2019][2937798.138032] .exec_clock : 0.000000 [Fri Apr 12 11:14:36 2019][2937798.143431] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:36 2019][2937798.148830] .min_vruntime : 99757454.764754 [Fri Apr 12 11:14:36 2019][2937798.154838] .max_vruntime : 0.000001 [Fri Apr 12 11:14:36 2019][2937798.160236] .spread : 0.000000 [Fri Apr 12 11:14:36 2019][2937798.165644] .spread0 : -17087896.840362 [Fri Apr 12 11:14:36 2019][2937798.171764] .nr_spread_over : 0 [Fri Apr 12 11:14:36 2019][2937798.176556] .nr_running : 2 [Fri Apr 12 11:14:36 2019][2937798.181347] .load : 2048 [Fri Apr 12 11:14:36 2019][2937798.186401] .runnable_load_avg : 0 [Fri Apr 12 11:14:36 2019][2937798.191192] .blocked_load_avg : 465 [Fri Apr 12 11:14:36 2019][2937798.196159] .tg_load_avg : 0 [Fri Apr 12 11:14:36 2019][2937798.200951] .tg_load_contrib : 0 [Fri Apr 12 11:14:36 2019][2937798.205743] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:36 2019][2937798.210538] .tg->runnable_avg : 0 [Fri Apr 12 11:14:36 2019][2937798.215330] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:36 2019][2937798.220124] .throttled : 0 [Fri Apr 12 11:14:36 2019][2937798.224917] .throttle_count : 0 [Fri Apr 12 11:14:36 2019][2937798.229709] .avg->runnable_avg_sum : 12508 [Fri Apr 12 11:14:36 2019][2937798.234848] .avg->runnable_avg_period : 46902 [Fri Apr 12 11:14:36 2019][2937798.240004] [Fri Apr 12 11:14:36 2019][2937798.241677] rt_rq[44]:/ [Fri Apr 12 11:14:36 2019][2937798.244302] .rt_nr_running : 0 [Fri Apr 12 11:14:36 2019][2937798.249096] .rt_throttled : 0 [Fri Apr 12 11:14:36 2019][2937798.253890] .rt_time : 0.000000 [Fri Apr 12 11:14:36 2019][2937798.259288] .rt_runtime : 1000.000000 [Fri Apr 12 11:14:36 2019][2937798.264948] [Fri Apr 12 11:14:36 2019][2937798.266620] runnable tasks: [Fri Apr 12 11:14:36 2019][2937798.269592] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:36 2019][2937798.280539] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:36 2019][2937798.291428] watchdog/44 231 -11.987747 734435 0 0.000000 21403.224184 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.302760] migration/44 232 0.000000 23794 0 0.000000 249597.651868 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.314097] ksoftirqd/44 233 99757465.189045 21658215 120 0.000000 305939.895850 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.325432] kworker/44:0H 235 3739.962538 9 100 0.000000 0.030509 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.336789] ccp-5-q2 13560 2819.349537 2 120 0.000000 0.009218 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.348132] kworker/44:1H 16836 99757474.112879 18298923 100 0.000000 133690.895571 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.359503] kiblnd_sd_00_01 117116 99757487.891269 2269178001 120 0.000000 19920911.085173 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.371014] Rkiblnd_sd_00_03 117118 99757489.556669 2268971653 120 0.000000 19925021.252891 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.382532] jbd2/dm-1-8 117247 85832783.665810 26 120 0.000000 0.384748 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.393955] ldlm_bl_02 117279 99757473.408485 1066468 120 0.000000 139640.195627 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.405452] mdt00_038 118247 99754269.638663 22952429 120 0.000000 1356829.313904 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.416892] mdt00_057 118305 83817546.880240 1251808 120 0.000000 45611.731462 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.428329] mdt00_068 118394 99670664.040136 34427041 120 0.000000 1867613.221595 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.439752] mdt00_073 118414 99684788.578202 30012118 120 0.000000 1707953.617219 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.451182] mdt_io00_005 118425 99748513.237015 1663779 120 0.000000 134271.956968 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.462605] mdt00_083 118451 96578554.988404 24791416 120 0.000000 1063400.184220 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.474035] mdt00_092 118527 90716554.977595 23455147 120 0.000000 895163.990591 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.485460] mdt00_101 118555 99757495.820924 27245083 120 0.000000 1595713.473496 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.496898] mdt00_102 118557 99687881.556486 19262620 120 0.000000 1073423.101168 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.508340] mdt_io00_032 118703 99710472.378422 2290957 120 0.000000 125730.857851 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.519770] mdt_io00_046 118800 99757482.314716 43588 120 0.000000 3707.307099 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.531196] mdt_io00_048 118812 99710262.996290 2394763 120 0.000000 192649.004705 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.542637] mdt_io00_071 118928 99757498.841891 3751279 120 0.000000 266307.444039 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.554068] mdt_io00_094 119016 85833306.488511 34 120 0.000000 2.122349 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.565491] mdt_io00_099 119033 85833305.014479 26 120 0.000000 1.054351 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.576915] mdt_io00_105 119064 99710262.996300 4168844 120 0.000000 330779.781979 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.588348] mdt_rdpg00_032 119149 99757504.360846 19342480 120 0.000000 898334.601969 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.599839] kworker/44:2 36816 98933351.796023 37659 120 0.000000 1365.711326 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.611173] kworker/44:1 43922 99757455.279691 14379 120 0.000000 1904.694606 0.000000 0 / [Fri Apr 12 11:14:36 2019][2937798.622523] [Fri Apr 12 11:14:36 2019][2937798.624213] cpu#45, 1996.203 MHz [Fri Apr 12 11:14:36 2019][2937798.627627] .nr_running : 0 [Fri Apr 12 11:14:36 2019][2937798.632438] .load : 1024 [Fri Apr 12 11:14:36 2019][2937798.637495] .nr_switches : 19224346855 [Fri Apr 12 11:14:36 2019][2937798.643155] .nr_load_updates : 1962416586 [Fri Apr 12 11:14:36 2019][2937798.648729] .nr_uninterruptible : 84327 [Fri Apr 12 11:14:36 2019][2937798.653868] .next_balance : 7232.399399 [Fri Apr 12 11:14:36 2019][2937798.659545] .curr->pid : 0 [Fri Apr 12 11:14:36 2019][2937798.664337] .clock : 2937798664.175439 [Fri Apr 12 11:14:36 2019][2937798.670518] .cpu_load[0] : 3072 [Fri Apr 12 11:14:36 2019][2937798.675571] .cpu_load[1] : 821 [Fri Apr 12 11:14:36 2019][2937798.680544] .cpu_load[2] : 2175 [Fri Apr 12 11:14:36 2019][2937798.685600] .cpu_load[3] : 1257 [Fri Apr 12 11:14:36 2019][2937798.690662] .cpu_load[4] : 862 [Fri Apr 12 11:14:36 2019][2937798.695655] .avg_idle : 238183 [Fri Apr 12 11:14:36 2019][2937798.700886] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:36 2019][2937798.706199] [Fri Apr 12 11:14:36 2019][2937798.707868] cfs_rq[45]:/ [Fri Apr 12 11:14:36 2019][2937798.710582] .exec_clock : 0.000000 [Fri Apr 12 11:14:36 2019][2937798.715980] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:36 2019][2937798.721379] .min_vruntime : 123977782.257805 [Fri Apr 12 11:14:36 2019][2937798.727472] .max_vruntime : 0.000001 [Fri Apr 12 11:14:36 2019][2937798.732872] .spread : 0.000000 [Fri Apr 12 11:14:36 2019][2937798.738271] .spread0 : 7132314.825103 [Fri Apr 12 11:14:36 2019][2937798.744190] .nr_spread_over : 0 [Fri Apr 12 11:14:36 2019][2937798.748993] .nr_running : 0 [Fri Apr 12 11:14:36 2019][2937798.753787] .load : 1024 [Fri Apr 12 11:14:36 2019][2937798.758838] .runnable_load_avg : 318 [Fri Apr 12 11:14:36 2019][2937798.763813] .blocked_load_avg : 136 [Fri Apr 12 11:14:36 2019][2937798.768779] .tg_load_avg : 0 [Fri Apr 12 11:14:36 2019][2937798.773581] .tg_load_contrib : 0 [Fri Apr 12 11:14:36 2019][2937798.778373] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:36 2019][2937798.783176] .tg->runnable_avg : 0 [Fri Apr 12 11:14:36 2019][2937798.787967] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:36 2019][2937798.792760] .throttled : 0 [Fri Apr 12 11:14:36 2019][2937798.797554] .throttle_count : 0 [Fri Apr 12 11:14:36 2019][2937798.802346] .avg->runnable_avg_sum : 32242 [Fri Apr 12 11:14:36 2019][2937798.807486] .avg->runnable_avg_period : 46811 [Fri Apr 12 11:14:36 2019][2937798.812633] [Fri Apr 12 11:14:36 2019][2937798.814308] rt_rq[45]:/ [Fri Apr 12 11:14:36 2019][2937798.816931] .rt_nr_running : 0 [Fri Apr 12 11:14:36 2019][2937798.821726] .rt_throttled : 0 [Fri Apr 12 11:14:36 2019][2937798.826517] .rt_time : 0.000000 [Fri Apr 12 11:14:36 2019][2937798.831918] .rt_runtime : 950.000000 [Fri Apr 12 11:14:36 2019][2937798.837490] [Fri Apr 12 11:14:36 2019][2937798.839162] runnable tasks: [Fri Apr 12 11:14:36 2019][2937798.842134] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:36 2019][2937798.853082] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:36 2019][2937798.863979] watchdog/45 236 -11.987015 734435 0 0.000000 27267.752636 0.000000 1 / [Fri Apr 12 11:14:36 2019][2937798.875329] migration/45 237 0.000000 31030 0 0.000000 355468.724868 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.886710] ksoftirqd/45 238 123977769.420635 14824610 120 0.000000 302682.389584 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.898046] kworker/45:0H 240 6406.127540 9 100 0.000000 0.028844 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.909430] kworker/45:1H 17624 123977834.953938 10139389 100 0.000000 71374.878279 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.920796] kworker/45:2 115197 123977693.296628 1989073 120 0.000000 62290.159023 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.932227] kworker/45:0 120679 101001495.929648 2 120 0.000000 0.010028 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.943669] ptlrpcd_01_02 117148 123973905.215476 652397 120 0.000000 16565.659502 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.955100] ptlrpcd_01_07 117153 123976641.506314 674067 120 0.000000 15739.073887 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.966530] mdt01_002 117468 123847321.952245 28156192 120 0.000000 1827965.503285 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.977960] mdt_out01_001 117495 123976015.121069 51286 120 0.000000 1535.165598 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937798.989438] osp-pre-40-0 117779 123977359.655861 39266 120 0.000000 1075.017292 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.000874] mdt01_011 118151 123977849.576510 22140697 120 0.000000 1597698.715829 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.012303] mdt01_017 118170 115955904.431828 6993668 120 0.000000 419209.186751 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.023733] mdt01_018 118173 123977857.277060 22457311 120 0.000000 1593625.879042 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.035174] mdt01_023 118189 113323109.019217 10015075 120 0.000000 400472.246710 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.046597] mdt01_025 118199 123857387.991636 31495424 120 0.000000 1937544.820758 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.058023] mdt01_036 118242 110451356.081125 1161434 120 0.000000 49988.862866 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.069449] mdt01_038 118250 123963693.509503 24647225 120 0.000000 1606104.282274 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.080875] mdt01_043 118265 123977856.124602 26746650 120 0.000000 1794159.784802 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.092307] mdt01_061 118315 123972561.882073 22783369 120 0.000000 1567034.809875 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.103737] mdt01_063 118321 123862604.613799 26385620 120 0.000000 1753385.730184 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.115176] mdt01_069 118331 123973794.979147 27749952 120 0.000000 1838395.834612 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.126599] mdt01_072 118334 123977869.600196 25355397 120 0.000000 1724416.146344 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.138042] mdt01_079 118342 109255265.172839 844509 120 0.000000 28264.448628 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.149463] mdt01_082 118349 123977872.516319 26174043 120 0.000000 1796603.035139 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.160913] mdt_io01_006 118445 123976825.919802 1271206 120 0.000000 99283.627976 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.172333] mdt01_110 118448 114018684.331890 11594840 120 0.000000 517342.501134 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.183787] mdt_io01_015 118666 123952054.287364 2767320 120 0.000000 216426.630027 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.195215] mdt_io01_016 118688 123813106.107873 1880508 120 0.000000 161311.466546 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.206652] mdt_io01_025 118754 123900852.901183 2972901 120 0.000000 227777.344001 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.218087] mdt_io01_037 118835 123813106.107935 1425186 120 0.000000 115681.857687 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.229532] mdt_io01_044 118870 123813106.107353 3645994 120 0.000000 250986.812524 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.240970] mdt_io01_053 118940 123813106.107984 1433017 120 0.000000 129850.213235 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.252444] mdt_io01_057 118964 123813106.107914 3694904 120 0.000000 249256.961679 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.263869] mdt_io01_063 118982 123952054.166447 2020083 120 0.000000 159012.091823 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.275303] R mdt_io01_071 119025 123977878.636704 2064501 120 0.000000 181524.199922 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.286742] mdt_rdpg01_015 119066 123976298.172525 28118709 120 0.000000 1290773.369536 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.298178] mdt_io01_081 119070 123907398.544012 1142602 120 0.000000 102102.123726 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.309599] mdt_io01_082 119072 123977867.388035 2155558 120 0.000000 153594.589390 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.321035] mdt_rdpg01_020 119085 123976298.172906 22451121 120 0.000000 1192391.003669 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.332456] mdt_rdpg01_025 119098 123977875.162449 26544042 120 0.000000 1297901.778223 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.343880] mdt_io01_088 119114 110445553.007884 53 120 0.000000 2.269665 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.355306] mdt_io01_096 119145 123977873.525355 438389 120 0.000000 35933.225983 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.366737] mdt_rdpg01_039 119174 123977892.851249 26599774 120 0.000000 1303375.039177 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.378163] mdt_io01_102 119182 110445553.366262 657352 120 0.000000 23056.838547 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.389592] mdt_rdpg01_053 119216 123976298.172295 23114152 120 0.000000 1327752.064597 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.401019] mdt_rdpg01_055 119225 123977712.151233 23743279 120 0.000000 1170338.521154 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.412451] mdt_rdpg01_059 119245 123977881.145876 30404887 120 0.000000 1426687.599322 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.423904] ldlm_cn01_009 120936 123977901.874308 51434798 120 0.000000 1294074.454232 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.435332] ldlm_cn01_012 122661 127707176.182039 49473106 120 0.000000 1259301.065910 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.446759] ldlm_cn01_024 126534 123977752.021770 45183386 120 0.000000 1172472.411413 0.000000 1 / [Fri Apr 12 11:14:37 2019][2937799.458309] [Fri Apr 12 11:14:37 2019][2937799.459983] cpu#46, 1996.203 MHz [Fri Apr 12 11:14:37 2019][2937799.463399] .nr_running : 1 [Fri Apr 12 11:14:37 2019][2937799.468198] .load : 0 [Fri Apr 12 11:14:37 2019][2937799.473005] .nr_switches : 11108313948 [Fri Apr 12 11:14:37 2019][2937799.478670] .nr_load_updates : 1695745188 [Fri Apr 12 11:14:37 2019][2937799.484253] .nr_uninterruptible : 70018 [Fri Apr 12 11:14:37 2019][2937799.489443] .next_balance : 7232.400237 [Fri Apr 12 11:14:37 2019][2937799.495118] .curr->pid : 117702 [Fri Apr 12 11:14:37 2019][2937799.500355] .clock : 2937799500.327344 [Fri Apr 12 11:14:37 2019][2937799.506542] .cpu_load[0] : 0 [Fri Apr 12 11:14:37 2019][2937799.511345] .cpu_load[1] : 328 [Fri Apr 12 11:14:37 2019][2937799.516320] .cpu_load[2] : 252 [Fri Apr 12 11:14:37 2019][2937799.521304] .cpu_load[3] : 448 [Fri Apr 12 11:14:37 2019][2937799.526277] .cpu_load[4] : 335 [Fri Apr 12 11:14:37 2019][2937799.531253] .avg_idle : 765393 [Fri Apr 12 11:14:37 2019][2937799.536488] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:37 2019][2937799.541808] [Fri Apr 12 11:14:37 2019][2937799.543485] cfs_rq[46]:/ [Fri Apr 12 11:14:37 2019][2937799.546197] .exec_clock : 0.000000 [Fri Apr 12 11:14:37 2019][2937799.551598] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:37 2019][2937799.557004] .min_vruntime : 83403343.500770 [Fri Apr 12 11:14:37 2019][2937799.563019] .max_vruntime : 0.000001 [Fri Apr 12 11:14:37 2019][2937799.568419] .spread : 0.000000 [Fri Apr 12 11:14:37 2019][2937799.573818] .spread0 : -33442184.840196 [Fri Apr 12 11:14:37 2019][2937799.579912] .nr_spread_over : 0 [Fri Apr 12 11:14:37 2019][2937799.584703] .nr_running : 0 [Fri Apr 12 11:14:37 2019][2937799.589497] .load : 0 [Fri Apr 12 11:14:37 2019][2937799.594300] .runnable_load_avg : 0 [Fri Apr 12 11:14:37 2019][2937799.599099] .blocked_load_avg : 316 [Fri Apr 12 11:14:37 2019][2937799.604075] .tg_load_avg : 0 [Fri Apr 12 11:14:37 2019][2937799.608866] .tg_load_contrib : 0 [Fri Apr 12 11:14:37 2019][2937799.613660] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:37 2019][2937799.618453] .tg->runnable_avg : 0 [Fri Apr 12 11:14:37 2019][2937799.623247] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:37 2019][2937799.628036] .throttled : 0 [Fri Apr 12 11:14:37 2019][2937799.632831] .throttle_count : 0 [Fri Apr 12 11:14:37 2019][2937799.637624] .avg->runnable_avg_sum : 7410 [Fri Apr 12 11:14:37 2019][2937799.641064] LustreError: dumping log to /tmp/lustre-log.1555092877.118709 [Fri Apr 12 11:14:37 2019][2937799.649636] .avg->runnable_avg_period : 46107 [Fri Apr 12 11:14:37 2019][2937799.654793] [Fri Apr 12 11:14:37 2019][2937799.656466] rt_rq[46]:/ [Fri Apr 12 11:14:37 2019][2937799.659091] .rt_nr_running : 0 [Fri Apr 12 11:14:37 2019][2937799.663884] .rt_throttled : 0 [Fri Apr 12 11:14:37 2019][2937799.668685] .rt_time : 0.000000 [Fri Apr 12 11:14:37 2019][2937799.674086] .rt_runtime : 950.000000 [Fri Apr 12 11:14:37 2019][2937799.679658] [Fri Apr 12 11:14:37 2019][2937799.681330] runnable tasks: [Fri Apr 12 11:14:37 2019][2937799.684302] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:37 2019][2937799.695248] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:37 2019][2937799.706140] watchdog/46 241 -11.986865 734436 0 0.000000 17967.511610 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.717479] migration/46 242 0.000000 16707 0 0.000000 120164.824798 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.728815] ksoftirqd/46 243 83403336.098826 19630394 120 0.000000 701918.340270 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.740169] kworker/46:0H 245 4543.278454 10 100 0.000000 0.044083 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.751539] ccp-3-q3 13526 2970.674756 2 120 0.000000 0.008998 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.762909] kworker/46:1H 21098 83403335.969520 7170579 100 0.000000 39119.815573 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.774292] osp-syn-11-2 117578 83403327.534909 235701 120 0.000000 6874.285166 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.785721] osp-syn-41-2 117614 83402795.375395 235918 120 0.000000 6322.928202 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.797147] osp-syn-6-2 117630 83402795.262481 238538 120 0.000000 6668.822744 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.808579] osp-syn-15-2 117638 83403333.772040 239152 120 0.000000 6560.360963 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.820005] ldlm_cn02_003 117684 83403327.061580 44105337 120 0.000000 1066679.186044 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.831457] mdt_rdpg02_002 118027 83391455.934608 25081722 120 0.000000 1218282.188151 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.842889] mdt02_015 118188 83403255.675960 21508033 120 0.000000 1504180.581368 0.000000 2 / [Fri Apr 12 11:14:37 2019][2937799.854319] mdt02_029 118262 79569785.172737 5584868 120 0.000000 400252.779749 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.865753] mdt02_041 118347 83403389.102635 21959936 120 0.000000 1318117.940810 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.877177] R mdt02_046 118367 83403386.331455 19836640 120 0.000000 1384753.496364 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.888604] mdt02_058 118447 83336847.402680 19442609 120 0.000000 1287191.786376 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.900048] mdt02_060 118460 83400486.372882 12588474 120 0.000000 1134252.269496 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.911479] mdt02_063 118467 73210940.478380 3207451 120 0.000000 127646.752098 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.922921] mdt02_064 118470 83393799.169556 15217318 120 0.000000 983009.996773 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.934376] mdt02_066 118473 83400994.961125 13873348 120 0.000000 1098345.283526 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.945808] mdt02_070 118483 83346669.030752 27003817 120 0.000000 1689994.505128 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.957234] mdt02_087 118522 83403255.750693 17616122 120 0.000000 1134385.304620 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.968662] mdt02_090 118530 83400651.595685 14521691 120 0.000000 1008410.911713 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.980086] mdt02_095 118541 83403345.727566 12079394 120 0.000000 902673.457211 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937799.991513] mdt02_106 118584 83345688.255994 16156227 120 0.000000 1187916.377473 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.002931] mdt_io02_007 118597 83354608.412627 680361 120 0.000000 44760.166463 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.014357] mdt_io02_012 118619 73705243.086220 607297 120 0.000000 25157.148297 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.025779] mdt_io02_017 118642 83379170.658528 2542619 120 0.000000 198445.121801 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.037211] mdt_io02_040 118718 73705241.675588 308913 120 0.000000 14726.995676 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.048634] mdt_io02_044 118742 82896180.399525 189203 120 0.000000 9270.981673 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.060056] mdt_io02_053 118772 73705241.974744 224146 120 0.000000 10629.583001 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.071480] mdt_io02_060 118806 82890449.395784 1722084 120 0.000000 112010.150490 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.082902] mdt_io02_067 118826 73705241.758593 65 120 0.000000 3.722277 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.094328] mdt_io02_070 118849 73705243.153874 652373 120 0.000000 27264.445250 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.105750] mdt_io02_075 118874 83379170.658398 1612178 120 0.000000 161588.411591 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.117181] mdt_io02_095 118939 83403394.935612 508845 120 0.000000 37993.237939 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.128604] mdt_io02_101 118957 83379170.658138 2451341 120 0.000000 204060.099447 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.140050] mdt_rdpg02_031 119131 83402621.933564 25833514 120 0.000000 1111590.412517 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.151478] mdt_rdpg02_043 119171 83396422.049140 24152168 120 0.000000 1190255.351563 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.162913] mdt_rdpg02_056 119239 83396422.050222 22683457 120 0.000000 1010878.068003 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.174342] ldlm_cn02_008 119914 83403327.062281 43209629 120 0.000000 1065862.192836 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.185775] kworker/46:2 125091 83403393.783230 232881 120 0.000000 11289.036725 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.197205] ldlm_cn02_018 129940 83403393.892781 36533045 120 0.000000 918542.278745 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.208648] osp-pre-41-1 2437 83402954.233506 27708 120 0.000000 719.187288 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.219990] ldlm_cn02_023 6478 83403402.338579 30393260 120 0.000000 788795.792714 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.231329] ldlm_cn02_025 6480 83403327.061610 32439824 120 0.000000 841626.438282 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.242666] ldlm_cn02_027 6493 83403403.442470 33511473 120 0.000000 848876.537764 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.254001] ldlm_cn02_029 6495 83403327.427244 28302319 120 0.000000 752028.343215 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.265341] kworker/46:1 24867 76812749.535618 2 120 0.000000 0.010059 0.000000 2 / [Fri Apr 12 11:14:38 2019][2937800.276740] [Fri Apr 12 11:14:38 2019][2937800.278420] cpu#47, 1996.203 MHz [Fri Apr 12 11:14:38 2019][2937800.281828] .nr_running : 0 [Fri Apr 12 11:14:38 2019][2937800.286630] .load : 0 [Fri Apr 12 11:14:38 2019][2937800.291432] .nr_switches : 11652352134 [Fri Apr 12 11:14:38 2019][2937800.297097] .nr_load_updates : 1588629834 [Fri Apr 12 11:14:38 2019][2937800.302672] .nr_uninterruptible : 28026 [Fri Apr 12 11:14:38 2019][2937800.307818] .next_balance : 7232.401052 [Fri Apr 12 11:14:38 2019][2937800.313480] .curr->pid : 0 [Fri Apr 12 11:14:38 2019][2937800.318279] .clock : 2937800318.117418 [Fri Apr 12 11:14:38 2019][2937800.324459] .cpu_load[0] : 0 [Fri Apr 12 11:14:38 2019][2937800.329260] .cpu_load[1] : 320 [Fri Apr 12 11:14:38 2019][2937800.334228] .cpu_load[2] : 70 [Fri Apr 12 11:14:38 2019][2937800.339113] .cpu_load[3] : 52 [Fri Apr 12 11:14:38 2019][2937800.343994] .cpu_load[4] : 56 [Fri Apr 12 11:14:38 2019][2937800.348883] .avg_idle : 616269 [Fri Apr 12 11:14:38 2019][2937800.354119] .max_idle_balance_cost : 5000000 [Fri Apr 12 11:14:38 2019][2937800.359466] [Fri Apr 12 11:14:38 2019][2937800.361143] cfs_rq[47]:/ [Fri Apr 12 11:14:38 2019][2937800.363854] .exec_clock : 0.000000 [Fri Apr 12 11:14:38 2019][2937800.369256] .MIN_vruntime : 0.000001 [Fri Apr 12 11:14:38 2019][2937800.374653] .min_vruntime : 64566269.861311 [Fri Apr 12 11:14:38 2019][2937800.380659] .max_vruntime : 0.000001 [Fri Apr 12 11:14:38 2019][2937800.386059] .spread : 0.000000 [Fri Apr 12 11:14:38 2019][2937800.391458] .spread0 : -52279331.581475 [Fri Apr 12 11:14:38 2019][2937800.397551] .nr_spread_over : 0 [Fri Apr 12 11:14:38 2019][2937800.402344] .nr_running : 0 [Fri Apr 12 11:14:38 2019][2937800.407138] .load : 0 [Fri Apr 12 11:14:38 2019][2937800.411928] .runnable_load_avg : 0 [Fri Apr 12 11:14:38 2019][2937800.416723] .blocked_load_avg : 64 [Fri Apr 12 11:14:38 2019][2937800.421602] .tg_load_avg : 0 [Fri Apr 12 11:14:38 2019][2937800.426393] .tg_load_contrib : 0 [Fri Apr 12 11:14:38 2019][2937800.431187] .tg_runnable_contrib : 0 [Fri Apr 12 11:14:38 2019][2937800.435980] .tg->runnable_avg : 0 [Fri Apr 12 11:14:38 2019][2937800.440772] .tg->cfs_bandwidth.timer_active: 0 [Fri Apr 12 11:14:38 2019][2937800.445565] .throttled : 0 [Fri Apr 12 11:14:38 2019][2937800.450358] .throttle_count : 0 [Fri Apr 12 11:14:38 2019][2937800.455153] .avg->runnable_avg_sum : 4855 [Fri Apr 12 11:14:38 2019][2937800.460203] .avg->runnable_avg_period : 46757 [Fri Apr 12 11:14:38 2019][2937800.465350] [Fri Apr 12 11:14:38 2019][2937800.467024] rt_rq[47]:/ [Fri Apr 12 11:14:38 2019][2937800.469651] .rt_nr_running : 0 [Fri Apr 12 11:14:38 2019][2937800.474445] .rt_throttled : 0 [Fri Apr 12 11:14:38 2019][2937800.479237] .rt_time : 0.000000 [Fri Apr 12 11:14:38 2019][2937800.484636] .rt_runtime : 950.000000 [Fri Apr 12 11:14:38 2019][2937800.490209] [Fri Apr 12 11:14:38 2019][2937800.491882] runnable tasks: [Fri Apr 12 11:14:38 2019][2937800.494854] task PID tree-key switches prio wait-time sum-exec sum-sleep [Fri Apr 12 11:14:38 2019][2937800.505799] ---------------------------------------------------------------------------------------------------------- [Fri Apr 12 11:14:38 2019][2937800.516691] watchdog/47 246 -11.987086 734436 0 0.000000 10358.223657 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.528030] migration/47 247 0.000000 11729 0 0.000000 27513.406319 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.539366] ksoftirqd/47 248 64566254.529661 3944073 120 0.000000 58645.532024 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.550703] kworker/47:0H 250 5884.729509 10 100 0.000000 0.032683 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.562045] kauditd 421 64564277.480281 391168 120 0.000000 3935.451434 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.573388] kworker/47:1 7314 64566254.527506 1471280 120 0.000000 25038.928258 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.584739] kdmwork-253:1 17187 64565966.589233 569323 120 0.000000 11799.240782 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.596075] kdmwork-253:2 17262 64566264.322949 6837689 120 0.000000 190244.156966 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.607417] dbus-daemon 18028 64564279.235409 2144197 120 0.000000 251470.129815 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.618762] in:imjournal 19305 64566257.837225 3522917 120 0.000000 109094.387800 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.630100] master 19681 64563996.346475 173161 120 0.000000 6585.970577 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.641442] kworker/47:1H 21549 64566254.520699 2537067 100 0.000000 11686.339395 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.652801] ptlrpcd_03_04 117174 64566258.343319 2289595 120 0.000000 53451.102211 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.664224] ldlm_cn03_002 117277 64564949.218097 49663172 120 0.000000 950949.652049 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.675645] ldlm_bl_01 117278 64566229.075394 1066042 120 0.000000 113351.867450 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.687071] mdt03_001 117473 64217826.708499 733164 120 0.000000 46616.045055 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.698496] mdt03_003 117506 64564977.379487 6333151 120 0.000000 279063.998526 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.709918] mdt03_005 117549 53446950.683391 1538243 120 0.000000 55237.947709 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.721337] mdt03_006 117559 64217826.708560 17872344 120 0.000000 964814.813618 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.732759] osp-pre-12-2 117569 64565972.402638 40175 120 0.000000 856.501767 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.744187] osp-pre-5-2 117605 64566268.913851 40969 120 0.000000 988.136270 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.755610] osp-pre-15-2 117639 64566188.898696 39938 120 0.000000 934.571449 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.767030] osp-syn-36-2 117652 64566231.572497 235455 120 0.000000 6602.925535 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.778453] ldlm_cn03_004 117689 64564949.218838 48301466 120 0.000000 926799.003191 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.789882] osp-syn-17-0 117752 64548580.725108 76033 120 0.000000 2267.478637 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.801307] osp-pre-0-0 117755 64566207.973494 38841 120 0.000000 792.222533 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.812728] osp-syn-44-0 117758 64548403.786999 75265 120 0.000000 2192.124464 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.824154] osp-pre-41-0 117767 64565934.415329 40156 120 0.000000 867.910335 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.835574] osp-pre-33-0 117773 64565980.431619 39348 120 0.000000 900.282604 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.846997] osp-syn-6-0 117782 64548352.573747 75610 120 0.000000 2202.821364 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.858422] osp-pre-22-0 117797 64565948.009475 39282 120 0.000000 789.164782 0.000000 3 / [Fri Apr 12 11:14:38 2019][2937800.869842] osp-pre-27-0 117801 64565934.416291 41050 120 0.000000 1111.877681 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.881285] mdt03_017 118239 64564998.702088 12333281 120 0.000000 898711.669344 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.892709] mdt03_024 118298 64566286.481508 21202598 120 0.000000 1175988.689656 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.904131] mdt03_026 118316 64217826.708579 18855058 120 0.000000 1032626.221636 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.915562] mdt03_040 118424 64564977.378746 22853751 120 0.000000 1183735.344650 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.926987] mdt03_045 118457 53835249.223545 3483176 120 0.000000 128458.294336 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.938412] mdt_rdpg03_016 118523 64545884.294496 11751656 120 0.000000 539687.000098 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.949838] mdt_rdpg03_018 118528 64545876.776808 16403363 120 0.000000 722660.268415 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.961265] mdt03_061 118579 64566295.268347 1900090 120 0.000000 121782.349956 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.972685] mdt03_064 118593 64217826.708249 13380098 120 0.000000 638014.710030 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.984109] mdt03_067 118610 64564998.702429 3705117 120 0.000000 238817.267360 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937800.995531] mdt03_070 118629 64564998.703320 2920655 120 0.000000 177203.647093 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.006958] mdt03_075 118660 64217659.958127 24078214 120 0.000000 1158595.783320 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.018385] mdt03_080 118678 64564977.379157 9537852 120 0.000000 689754.728735 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.029807] mdt_io03_020 118683 53965052.852696 38 120 0.000000 2.158798 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.041229] mdt03_082 118686 64217659.950663 712808 120 0.000000 44282.897177 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.052656] mdt03_090 118719 64217826.708910 13997325 120 0.000000 644497.278245 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.064079] mdt03_098 118753 64566295.003446 739619 120 0.000000 49047.206127 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.075498] mdt03_099 118755 64217659.761493 15700275 120 0.000000 1035696.758870 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.086924] mdt_io03_039 118769 53965052.703203 439766 120 0.000000 16357.484953 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.098346] mdt_io03_041 118775 53965096.662861 181600 120 0.000000 7419.829186 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.109771] mdt_io03_053 118821 53965053.831212 54 120 0.000000 2.839598 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.121200] mdt_io03_069 118872 53965127.077355 93 120 0.000000 3.585025 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.132684] ldlm_cn03_007 119782 64564949.226121 44632423 120 0.000000 859717.912401 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.144127] osp-pre-42-3 2251 64565982.302706 28040 120 0.000000 588.798247 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.155461] osp-pre-19-3 2259 64565961.419611 28046 120 0.000000 538.956900 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.166796] osp-pre-25-3 2263 64566171.163446 28022 120 0.000000 571.124395 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.178133] osp-syn-15-3 2302 64562465.690017 9527 120 0.000000 317.017377 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.189467] osp-pre-22-3 2309 64566283.275691 27851 120 0.000000 784.280250 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.200805] osp-pre-2-1 2382 64565971.765630 27367 120 0.000000 559.543435 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.212145] osp-pre-37-1 2473 64565963.951654 27609 120 0.000000 638.827684 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.223478] osp-pre-8-1 2477 64566298.379291 28046 120 0.000000 617.003008 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.234815] dist_txn-1 2507 64528655.399031 414 120 0.000000 29.546016 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.246150] mdt_io03_090 3060 53965052.338570 6 120 0.000000 0.368581 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.257497] kworker/47:0 44759 63984544.206537 2 120 0.000000 0.009848 0.000000 3 / [Fri Apr 12 11:14:39 2019][2937801.268850] [Fri Apr 12 11:14:39 2019][2937801.271086] INFO: rcu_sched detected stalls on CPUs/tasks: { 2 47} (detected by 47, t=6 jiffies, g=565156739, c=565156738, q=23839744) [Fri Apr 12 11:14:39 2019][2937801.283488] Task dump for CPU 47: [Fri Apr 12 11:14:39 2019][2937801.286979] swapper/47 R running task 0 0 1 0x00000008 [Fri Apr 12 11:14:39 2019][2937801.294259] Call Trace: [Fri Apr 12 11:14:39 2019][2937801.296888] [] sched_show_task+0xa8/0x110 [Fri Apr 12 11:14:39 2019][2937801.303362] [] dump_cpu_task+0x39/0x70 [Fri Apr 12 11:14:39 2019][2937801.308937] [] rcu_dump_cpu_stacks+0x90/0xd0 [Fri Apr 12 11:14:39 2019][2937801.315037] [] rcu_check_callbacks+0x6f2/0x730 [Fri Apr 12 11:14:39 2019][2937801.321313] [] ? tick_sched_do_timer+0x50/0x50 [Fri Apr 12 11:14:39 2019][2937801.327589] [] update_process_times+0x46/0x80 [Fri Apr 12 11:14:39 2019][2937801.333784] [] tick_sched_handle+0x30/0x70 [Fri Apr 12 11:14:39 2019][2937801.339702] [] tick_sched_timer+0x39/0x80 [Fri Apr 12 11:14:39 2019][2937801.345538] [] __hrtimer_run_queues+0xf3/0x270 [Fri Apr 12 11:14:39 2019][2937801.351810] [] hrtimer_interrupt+0xaf/0x1d0 [Fri Apr 12 11:14:39 2019][2937801.357818] [] local_apic_timer_interrupt+0x3b/0x60 [Fri Apr 12 11:14:39 2019][2937801.364516] [] smp_apic_timer_interrupt+0x43/0x60 [Fri Apr 12 11:14:39 2019][2937801.371044] [] apic_timer_interrupt+0x162/0x170 [Fri Apr 12 11:14:39 2019][2937801.377394] [] ? __cpuidle_text_start+0x8/0x8 [Fri Apr 12 11:14:39 2019][2937801.384206] [] ? native_safe_halt+0x6/0x10 [Fri Apr 12 11:14:39 2019][2937801.390126] [] default_idle+0x1e/0xc0 [Fri Apr 12 11:14:39 2019][2937801.395615] [] arch_cpu_idle+0x20/0xc0 [Fri Apr 12 11:14:39 2019][2937801.401186] [] cpu_startup_entry+0x14a/0x1e0 [Fri Apr 12 11:14:39 2019][2937801.407281] [] start_secondary+0x1f7/0x270 [Fri Apr 12 11:14:39 2019][2937801.413206] [] start_cpu+0x5/0x14